Friendly, beginner-to-advanced notes on the foundations of Machine Learning. Read it top to bottom the first time, then use the table of contents as a quick reference.

---

📑 Table of Contents

1. Introduction to AI, ML and Deep Learning
2. What Is Machine Learning?
3. History and Evolution of ML
4. Types of Machine Learning (Overview)
5. Supervised Learning
6. Unsupervised Learning
7. Semi-Supervised Learning
8. Self-Supervised Learning
9. Reinforcement Learning
10. Batch Learning vs Online Learning
11. Instance-Based vs Model-Based Learning
12. Applications and Limitations of ML
13. Machine Learning Project Lifecycle
14. Quick Revision Cheat Sheet

---

1. Introduction to Artificial Intelligence, Machine Learning and Deep Learning

These three terms get thrown around as if they mean the same thing. They don't. They are nested ideas, like Russian dolls.

┌──────────────────────────────────────────────┐
│  Artificial Intelligence (AI)                │
│  ┌────────────────────────────────────────┐  │
│  │  Machine Learning (ML)                 │  │
│  │  ┌──────────────────────────────────┐  │  │
│  │  │  Deep Learning (DL)              │  │  │
│  │  └──────────────────────────────────┘  │  │
│  └────────────────────────────────────────┘  │
└──────────────────────────────────────────────┘

🧠 Artificial Intelligence (AI)

The big goal: make machines do things that normally need human intelligence, such as reasoning, understanding language, recognising images, planning and making decisions.

AI is a goal, not a single technique. There are two broad ways to reach it:

Approach	How it works	Example
Rule-based (symbolic) AI	Humans write explicit rules: "IF this THEN that"	A chess program with hand-coded strategies, an expert system for diagnosis
Learning-based AI	The machine figures out the rules from data	A spam filter that learns from millions of emails

📊 Machine Learning (ML)

A subset of AI where systems learn patterns from data instead of being explicitly programmed with rules. You show the machine many examples, and it works out the pattern itself.

🕸️ Deep Learning (DL)

A subset of ML that uses artificial neural networks with many layers (hence "deep"). It is especially good at messy, unstructured data such as images, audio and text, because it can learn useful features on its own rather than needing a human to design them.

Comparing the three

Aspect	AI	ML	Deep Learning
Scope	Broadest	Subset of AI	Subset of ML
Core idea	Intelligent behaviour	Learn from data	Learn with many-layered neural networks
Needs data?	Not always	Yes	Usually a lot
Feature design	Hand-crafted rules	Often hand-crafted features	Learns features automatically
Compute needs	Varies	Moderate	High (often GPUs)
Example	Rule-based chatbot	Spam filter	Face recognition, speech-to-text

💡 Remember it like this: All deep learning is machine learning, and all machine learning is AI, but not all AI is machine learning.

---

2. What Is Machine Learning?

The simple definition

Machine Learning is the practice of building systems that improve at a task through experience (data), without being told step by step how to do it.

Two classic definitions worth remembering

• Arthur Samuel (1959): ML is the field of study that gives computers the ability to learn without being explicitly programmed.
• Tom Mitchell (1997), the more precise one: A program is said to learn from experience E with respect to some task T and performance measure P, if its performance at T, as measured by P, improves with experience E.

Example (spam filter):

• T (Task): flag spam emails
• E (Experience): a set of emails labelled "spam" or "not spam"
• P (Performance): percentage of emails classified correctly (accuracy)

Traditional programming vs Machine Learning

TRADITIONAL PROGRAMMING
  Data  ─┐
         ├──►  Program  ──►  Output
  Rules ─┘

MACHINE LEARNING
  Data   ─┐
          ├──►  Learning Algorithm  ──►  Rules (the "model")
  Output ─┘

In traditional programming, you write the rules. In ML, you give data and the answers, and the algorithm discovers the rules.

Why do we even need ML?

1. Rules are too complex to write by hand. Try writing exact rules to recognise a cat in any photo.
2. Rules keep changing. Spammers change tactics constantly; a learning system can adapt by retraining.
3. Problems are too large or messy for humans to hand-code (speech, language, huge datasets).
4. Insights from data. ML can reveal patterns humans would miss, helping us understand a problem better.

Core vocabulary you'll see everywhere

Term	Meaning
Dataset	The collection of examples used for learning
Instance / Sample / Example	One row in the dataset (e.g., one email, one house)
Feature (attribute)	An input property of an instance (e.g., size, location, word count)
Label / Target	The answer we want to predict (e.g., price, spam/not spam)
Model	The learned mathematical function that maps inputs to outputs
Training	The process of the algorithm learning from data
Training set	Data used to teach the model
Test set	Unseen data used to check how well the model learned
Prediction / Inference	Using the trained model on new data
Parameters	Values the model learns by itself during training
Hyperparameters	Settings you choose before training (e.g., learning rate)
Generalisation	How well the model performs on new, unseen data

A tiny sample dataset to picture it

Size (sq ft)	Bedrooms	Location score	Price (label)
1200	2	7	180,000
1800	3	8	290,000
950	1	5	120,000

Each row is an instance, the first three columns are features, and Price is the label.

---

3. History and Evolution of ML

ML has had a rollercoaster journey, with periods of excitement followed by disappointment ("AI winters"), and finally the current boom.

Timeline

Era	Milestone
1943	McCulloch & Pitts propose a mathematical model of an artificial neuron
1950	Alan Turing publishes "Computing Machinery and Intelligence" and proposes the Turing Test
1952	Arthur Samuel builds a checkers program that improves by playing
1956	The Dartmouth Conference coins the term "Artificial Intelligence"
1958	Frank Rosenblatt invents the Perceptron, an early neural network
1960s	Early pattern recognition and nearest-neighbour ideas appear
1969	Minsky & Papert show the limits of single-layer perceptrons, and neural network enthusiasm drops
1974–1980	First AI Winter: funding and interest decline
1980s	Expert systems boom; backpropagation is popularised (1986), reviving neural networks
1987–1993	Second AI Winter: expert systems prove brittle and expensive
1990s	Shift to data-driven ML: Support Vector Machines, decision trees, ensemble ideas, and practical applications such as spam filtering
1997	IBM's Deep Blue defeats chess champion Garry Kasparov
2006	Geoffrey Hinton and colleagues revive interest in deep learning
2011	IBM Watson wins Jeopardy!
2012	AlexNet wins the ImageNet competition by a large margin, triggering the deep learning boom
2014	Generative Adversarial Networks (GANs) are introduced
2016	DeepMind's AlphaGo defeats world Go champion Lee Sedol
2017	The Transformer architecture is introduced ("Attention Is All You Need")
2018 onwards	Large pre-trained language models and self-supervised learning take off
2020s	Generative AI and large foundation models reach the mainstream

Why did ML take off in the 2010s?

Three ingredients finally came together:

1. 📦 Big Data: the internet, smartphones and sensors produce massive datasets.
2. ⚡ Compute power: GPUs and cloud computing make heavy training affordable.
3. 🧪 Better algorithms and open tools: improved techniques and free libraries lowered the barrier to entry.

The big shifts in thinking

Rule-based systems  ──►  Statistical / classical ML  ──►  Deep Learning  ──►  Large pre-trained (foundation) models
 (hand-written rules)    (learn from hand-made features)  (learn features too)   (learn general knowledge, then adapt)

---

4. Types of Machine Learning

ML systems can be categorised in several different ways. These categories are not mutually exclusive: a single system can fit into more than one.

Way 1: By the amount and type of supervision during training

Type	Short idea
Supervised	Learn from labelled data
Unsupervised	Find structure in unlabelled data
Semi-supervised	A little labelled data + lots of unlabelled data
Self-supervised	The data creates its own labels
Reinforcement	Learn by trial, error and rewards

Way 2: By whether the system learns incrementally

• Batch learning vs Online learning (see Section 10)

Way 3: By how the system generalises

• Instance-based vs Model-based learning (see Section 11)

🧩 Example of combining categories: A modern spam filter can be supervised (trained on labelled emails), online (updates continuously), and model-based (learns a model rather than comparing to stored emails).

---

5. Supervised Learning

🎯 Core idea

The training data comes with the correct answers (labels). Like a student studying with an answer key, the model learns the relationship between inputs (features) and outputs (labels), then predicts labels for new data.

Features (X) + Labels (y)  ──►  Training  ──►  Model  ──►  Predictions for new X

The two main kinds of supervised tasks

A) Classification: predict a category

The output is a discrete class.

• Binary classification: two classes (spam / not spam, fraud / genuine, disease / no disease)
• Multi-class classification: more than two classes (digit 0–9, type of flower)
• Multi-label classification: one item can have several labels at once (a photo tagged beach, sunset and people)

B) Regression: predict a number

The output is a continuous value.

• Predicting house prices
• Forecasting temperature
• Estimating a person's age from a photo

	Classification	Regression
Output	Category / class	Continuous number
Question answered	"Which one?"	"How much?"
Example	Is this email spam?	What will this house sell for?

Common supervised algorithms (just names, so you can recognise them)

Linear Regression, Logistic Regression, k-Nearest Neighbours, Decision Trees, Random Forests, Support Vector Machines, Naive Bayes, Neural Networks.

Strengths and weaknesses

✅ Strengths	⚠️ Weaknesses
Clear, measurable goals	Needs labelled data, which can be costly and slow to get
Easy to evaluate against known answers	Quality of labels limits quality of the model
Very mature and widely used	Can't discover patterns it was never shown labels for

Real-world examples

Email spam detection, credit-card fraud detection, medical image diagnosis, loan default prediction, price prediction, handwriting recognition.

---

6. Unsupervised Learning

🔍 Core idea

The training data has no labels. The algorithm explores the data and tries to discover hidden structure by itself, like sorting a pile of unknown objects into groups without anyone telling you what the groups are.

Unlabelled data (X only)  ──►  Algorithm  ──►  Discovered structure (groups, patterns, simplified view)

Main kinds of unsupervised tasks

Task	What it does	Example
Clustering	Groups similar instances together	Segmenting customers by behaviour
Dimensionality reduction	Shrinks the number of features while keeping the important information	Compressing 100 features into 2–3 for visualisation
Anomaly / outlier detection	Spots instances that look very different from normal	Detecting unusual bank transactions or machine faults
Novelty detection	Spots new instances unlike anything in clean training data	Flagging a new kind of defect
Association rule learning	Finds relationships between items	"People who buy bread often buy butter" (market-basket analysis)

Common unsupervised algorithms (names only)

K-Means, Hierarchical Clustering, DBSCAN, Principal Component Analysis (PCA), t-SNE, Autoencoders, Isolation Forest, Apriori.

Strengths and weaknesses

✅ Strengths	⚠️ Weaknesses
No labelling cost	Hard to know if the results are "correct"
Can reveal unexpected patterns	Results can be subjective and need human interpretation
Great for exploring unfamiliar data	Output quality depends heavily on the algorithm and settings

Real-world examples

Customer segmentation, recommendation groupings, fraud and anomaly detection, data compression, visualising high-dimensional data, topic grouping of documents.

---

7. Semi-Supervised Learning

🧪 Core idea

Labelling data is expensive, but unlabelled data is usually cheap and plentiful. Semi-supervised learning uses a small amount of labelled data together with a large amount of unlabelled data.

Few labelled examples  ┐
                       ├──►  Model learns from both  ──►  Better than labelled data alone
Many unlabelled ones   ┘

An intuitive example

Your photo app groups all photos of the same person (an unsupervised step, by clustering faces). You then label that person once ("This is Maya"), and the app automatically tags her in all the other photos (a supervised step).

How it typically works

1. Train an initial model on the small labelled set.
2. Use it to predict labels for the unlabelled data (these are called pseudo-labels).
3. Keep the confident predictions and add them to the training set.
4. Retrain and repeat.

Why it's useful

• Cuts labelling effort and cost dramatically
• Often performs better than using only the few labels you have
• Practical in fields where expert labelling is expensive (medical scans, speech, legal documents)

Watch out

• If the initial model makes confident mistakes, those errors can be reinforced through pseudo-labels.
• It works best when the unlabelled data comes from the same kind of distribution as the labelled data.

Real-world examples

Medical image analysis, speech recognition, web page classification, photo-tagging services.

---

8. Self-Supervised Learning

🪞 Core idea

The system creates its own labels from the data itself, so no humans need to label anything. It hides or removes part of the input and trains the model to predict the missing part from the rest.

This is sometimes seen as a clever form of unsupervised learning, because the raw data is unlabelled, but the training itself looks like supervised learning (there's a target to predict, generated automatically).

Raw data ──► Hide a part ──► Model tries to predict the hidden part ──► Compare with the real part ──► Learn

Simple examples of "made-up" tasks (called pretext tasks)

Data type	Pretext task
Text	Hide a word in a sentence and predict it: "The cat sat on the ___"
Text	Predict the next word in a sequence
Images	Mask a patch of an image and reconstruct it
Images	Predict the rotation applied to a picture
Audio / Video	Predict missing or future segments

Why it matters

• Huge amounts of raw, unlabelled data (text, images, audio) are available for free.
• The model learns rich general representations of the world.
• These pre-trained models can then be fine-tuned on a small labelled dataset for a specific job. This is the recipe behind many modern large language and vision models.

Pre-training then fine-tuning

Step 1 (Self-supervised pre-training)       Step 2 (Supervised fine-tuning)
Massive unlabelled data ──► General model ──► Small labelled data ──► Specialised model

Comparing with its neighbours

	Supervised	Unsupervised	Self-supervised
Needs human labels?	Yes	No	No
Where do labels come from?	Humans	None	The data itself
Typical goal	Predict given labels	Find structure	Learn useful representations

---

9. Reinforcement Learning

🎮 Core idea

An agent learns by interacting with an environment, taking actions and receiving rewards (good) or penalties (bad). Over time it learns a policy, a strategy that maximises the total reward. It's like training a dog with treats, or learning a video game by playing it again and again.

              action
     ┌────────────────────┐
     │                    ▼
  ┌───────┐         ┌─────────────┐
  │ Agent │◄────────│ Environment │
  └───────┘ state + └─────────────┘
            reward

Key terms

Term	Meaning
Agent	The learner or decision-maker
Environment	The world the agent interacts with
State	The current situation the agent observes
Action	A move the agent can make
Reward	Feedback signal after an action (positive or negative)
Policy	The agent's strategy: which action to take in which state
Value	Expected total future reward from a state
Episode	One complete run from start to end of a task

The learning loop

1. Agent observes the current state.
2. It picks an action using its policy.
3. The environment gives back a reward and a new state.
4. The agent updates its policy to get more reward in future.
5. Repeat many, many times.

The big challenge: Exploration vs Exploitation

• Exploration: try new actions to discover potentially better strategies.
• Exploitation: use what you already know works to collect reward now.

A good agent balances both. Think of choosing a restaurant: do you go to your favourite (exploit) or try the new place (explore)?

What makes it different from supervised learning?

• There are no labelled correct answers, only reward signals.
• Rewards are often delayed. A move made now might only pay off many steps later.
• The agent's own actions change what data it sees next.

Real-world examples

Game-playing agents (chess, Go, video games), robot movement and control, self-driving decision-making, resource allocation, recommendation tuning, and fine-tuning language models using human feedback.

Strengths and weaknesses

✅ Strengths	⚠️ Weaknesses
Learns complex behaviour without labelled data	Needs a huge number of trial interactions
Can discover strategies humans haven't thought of	Designing a good reward is hard and a poorly designed one gets exploited
Suited to sequential decision-making	Training can be unstable and expensive

---

10. Batch Learning vs Online Learning

This category is about how and when the system learns: all at once, or little by little.

📦 Batch Learning (Offline Learning)

The model is trained using all the available data in one go. Once trained, it is deployed and stops learning. It just applies what it knows.

• To learn from new data, you must retrain from scratch on the old + new data, then replace the old model.
• This retraining can be automated and scheduled (daily, weekly, etc.).

Pros

• Simple to build and understand
• Stable and predictable behaviour
• Easier to test before deployment

Cons

• Needs lots of compute, time and memory for each full retraining
• Can't adapt quickly to fast-changing situations
• Struggles with truly huge datasets that don't fit in memory
• Performance can slowly decay as the real world changes (called model drift)

🌊 Online Learning (Incremental Learning)

The model is trained incrementally by feeding it data instances one by one or in small groups (mini-batches). It learns on the fly, and each learning step is fast and cheap.

Pros

• Adapts rapidly to new data and changing conditions
• Uses little memory because old data can be discarded after learning
• Handles huge datasets via out-of-core learning (loading chunks of data piece by piece)
• Suited to continuous data streams (stock prices, sensor data)

Cons

• Bad data can quickly harm the model (e.g., a faulty sensor or malicious inputs), so live monitoring is needed
• Harder to manage and debug
• Needs careful tuning of the learning rate

The learning rate in online learning

• High learning rate: adapts fast but quickly forgets older patterns.
• Low learning rate: learns slowly and is more stable but less responsive to new trends and more resistant to noise.

Side-by-side comparison

Feature	Batch Learning	Online Learning
Training data	Entire dataset at once	Streams, one at a time or small groups
Updates after deployment	Retrain from scratch	Continuous small updates
Adapts to change	Slowly	Quickly
Resource usage	High per retrain	Low per update
Handles huge data	Difficult	Good (out-of-core)
Risk	Model goes stale	Bad data can degrade it quickly
Typical use	Image classifier retrained monthly	Live stock or recommendation updates

🔔 Note: The word "online" here refers to learning incrementally, not to the internet. An online learner doesn't have to be connected to a network.

---

11. Instance-Based vs Model-Based Learning

This category is about how the system generalises, meaning how it makes predictions on data it has never seen.

🗃️ Instance-Based Learning

The system memorises the training examples and, for a new case, compares it to the stored ones using a similarity measure. It generalises by saying "this new item looks most like these known ones."

Example: A spam filter that flags a new email if it's very similar (shares many words) to emails already known to be spam. The classic algorithm is k-Nearest Neighbours (k-NN).

New point ──► Find the most similar stored examples ──► Predict based on those neighbours

Pros

• Simple and intuitive
• No explicit training phase (it "learns" by storing data)
• Can handle complex, irregular patterns

Cons

• Needs to store all the data, so memory-heavy
• Predictions can be slow since it compares against many stored examples
• Sensitive to irrelevant features and the choice of similarity measure

📐 Model-Based Learning

The system builds a model from the training examples and then uses the model to make predictions. It generalises by learning a general rule or function that summarises the data.

Example: Fit a line that relates house size to price, then use that line to predict prices for new houses (Linear Regression).

Training data ──► Choose a model type ──► Learn its parameters ──► Use the model to predict

The typical workflow of model-based learning

1. Study the data and pick a model type (e.g., a linear model).
2. Define a performance measure: either a utility/fitness function (how good the model is) or a cost/loss function (how bad the model is).
3. Train the model by finding the parameter values that minimise the cost.
4. Apply the model to make predictions on new data (inference).

A simple model might look like: price = θ₀ + θ₁ × size. Training finds the best values of θ₀ and θ₁.

Pros

• Fast predictions once trained (just apply the function)
• The training data doesn't need to be stored afterwards
• Often more compact and scalable

Cons

• Needs a training phase
• If the chosen model type is a poor fit for the data, results suffer

Comparison

Feature	Instance-Based	Model-Based
How it generalises	Compares new data to stored examples	Uses a learned function/model
What's stored	All (or many) training instances	Only the model parameters
Training phase	Minimal (just storing)	Explicit optimisation
Prediction speed	Slower with large data	Fast
Memory	High	Low
Example	k-Nearest Neighbours	Linear Regression, Neural Networks

---

12. Applications and Limitations of ML

🌍 Applications

Healthcare

• Detecting diseases from X-rays, MRIs and scans
• Predicting patient risk and treatment outcomes
• Speeding up drug discovery

Finance

• Fraud detection
• Credit scoring and loan approval
• Algorithmic trading and risk assessment

E-commerce and Entertainment

• Product and movie/music recommendations
• Dynamic pricing
• Customer segmentation and churn prediction

Transportation

• Self-driving and driver-assistance systems
• Route optimisation and traffic prediction
• Demand forecasting for ride services

Language and Speech

• Machine translation, chatbots and virtual assistants
• Speech-to-text and voice assistants
• Sentiment analysis and text summarisation

Computer Vision

• Face and object recognition
• Quality inspection in manufacturing
• Document and handwriting recognition (OCR)

Industry and Agriculture

• Predictive maintenance for machines
• Crop disease detection and yield prediction

Cybersecurity

• Intrusion and malware detection
• Spam and phishing filters

🚧 Limitations and Challenges

ML is powerful but not magic. Knowing its limits is just as important as knowing its strengths.

1. Data problems

Issue	What it means
Insufficient data	Models often need lots of examples to learn well; too few gives poor results
Non-representative data	If training data doesn't reflect the real world, the model won't generalise (sampling bias)
Poor quality data	Errors, noise, outliers and missing values lead to poor models
Irrelevant features	Useless inputs confuse the model; good features matter a lot
Label errors	In supervised learning, wrong labels teach wrong lessons

🗑️ "Garbage in, garbage out." A model is only as good as its data.

2. Model problems

• Overfitting: the model memorises the training data (including its noise) and performs poorly on new data. Like a student who memorised past answers but can't solve new questions.
• Underfitting: the model is too simple to capture the pattern and performs poorly even on training data.
• Poor generalisation: works in the lab, fails in the real world.

Underfitting          Good fit            Overfitting
(too simple)       (just right)         (too complex)
  Misses the         Captures the        Chases every
  pattern            real pattern        bit of noise

Ways to fight overfitting: get more data, simplify the model, add regularisation (constraining the model), and reduce noise in the data. Ways to fight underfitting: use a more powerful model, provide better features, reduce constraints on the model.

3. Ethical and social concerns

• Bias and fairness: models can inherit and even amplify biases present in data, leading to unfair outcomes.
• Privacy: training often involves personal or sensitive data.
• Accountability: who is responsible when an automated decision goes wrong?
• Misuse: the same technology can be used harmfully (e.g., deceptive fake content).

4. Practical and technical limits

• Interpretability ("black box" problem): complex models, especially deep ones, are hard to explain.
• Cost: collecting data, labelling it, and computing power can be expensive.
• Correlation, not causation: ML finds patterns, which doesn't mean it understands why they exist.
• Changing environments: models degrade when the real world drifts away from the training data.
• Brittleness: small, unusual changes to inputs can sometimes cause large errors.
• Not always the right tool: if a simple rule solves the problem, ML may be unnecessary overkill.

When should you use ML?

✅ ML is a good fit when	❌ ML is probably not needed when
Rules are too complex to hand-code	A simple, fixed rule works fine
The problem changes over time	The logic is stable and clearly known
You have enough relevant data	You have little or no data
Approximate answers are acceptable	You need guaranteed 100% exact results

---

13. Machine Learning Project Lifecycle

A real ML project is much more than "train a model." It's an iterative cycle, and you will often loop back to earlier steps.

 ┌─► 1. Define the Problem
 │        │
 │   2. Collect Data
 │        │
 │   3. Prepare & Explore Data
 │        │
 │   4. Feature Engineering
 │        │
 │   5. Select & Train Model(s)
 │        │
 │   6. Evaluate
 │        │
 │   7. Tune & Improve  ───────┐ (loop back if not good enough)
 │        │                    │
 │   8. Deploy                 │
 │        │                    │
 └── 9. Monitor & Maintain ◄───┘

Step 1: Define the Problem and Goals

Before touching data, get clarity.

• What's the business or real-world objective?
• What does success look like, and how will it be measured?
• Is ML even the right approach? Which type of ML fits (classification, regression, clustering...)?
• What are the constraints (time, cost, accuracy needed, interpretability)?
• How is the problem solved today? (This gives you a baseline to beat.)

Step 2: Data Collection

• Identify and gather data from relevant sources (databases, files, APIs, sensors, surveys).
• Make sure there's enough data and that it is relevant and representative.
• Consider legal and ethical issues such as privacy and consent.

Step 3: Data Preparation and Exploration (EDA)

Usually the most time-consuming step.

Exploration (Exploratory Data Analysis): understand your data via summaries, charts, distributions and correlations.

Cleaning and preprocessing:

• Handle missing values (remove, or fill in sensibly)
• Remove or treat outliers and duplicates
• Fix inconsistent formats and errors
• Encode categorical variables into numbers
• Scale/normalise numerical features where needed

Splitting the data: the golden rule is to never evaluate on data you trained on.

Split	Purpose	Typical share
Training set	Teach the model	~60–80%
Validation set	Compare models and tune settings	~10–20%
Test set	Final, honest check of performance (touch it only once at the end)	~10–20%

⚠️ Data leakage: accidentally letting information from the test set (or from the future) slip into training. It gives unrealistically great results that collapse in the real world.

Step 4: Feature Engineering

Features often matter more than which algorithm you choose.

• Feature selection: choose the most useful existing features.
• Feature extraction: combine features into more informative ones.
• Feature creation: build new features from domain knowledge (e.g., from a date, derive "day of week" or "is weekend").

Step 5: Model Selection and Training

• Start with a simple baseline model so you have something to compare against.
• Try several candidate models suited to your problem type.
• Train each on the training data.

Step 6: Evaluation

Measure how well the model does using suitable metrics on validation/test data.

Task	Common metrics
Classification	Accuracy, Precision, Recall, F1-score, Confusion Matrix, ROC-AUC
Regression	MAE (Mean Absolute Error), MSE / RMSE, R² score
Clustering	Silhouette score (and human judgement)

Key classification terms:

• Precision: of everything predicted positive, how much was truly positive?
• Recall: of everything truly positive, how much did we catch?
• F1-score: balance of precision and recall.

Also check overfitting vs underfitting by comparing training and validation performance. A large gap suggests overfitting.

Step 7: Tuning and Improvement

• Hyperparameter tuning (e.g., via grid search or random search)
• Better features, more or cleaner data
• Trying different or combined models
• Cross-validation for more reliable performance estimates
• Iterate: go back to earlier steps as needed

Step 8: Deployment

Put the model into the real world where it creates value.

• Integrate into an app, website, service or pipeline (e.g., exposed through an API)
• Decide on batch predictions (processed periodically) vs real-time predictions (instant)
• Ensure it runs reliably and at the required speed and scale

Step 9: Monitoring and Maintenance

Deployment is not the finish line.

• Track performance continuously in production.
• Watch for data drift (input data changes) and concept drift (the relationship between input and output changes).
• Retrain or update the model regularly with fresh data.
• Keep versions, logs and documentation so results are reproducible.
• Handle feedback, errors and edge cases.

Where does your learning type come in?

Your choice from earlier sections shapes the lifecycle: for example, batch systems schedule periodic retraining, while online systems build live monitoring and safeguards into the pipeline.

---

14. Quick Revision Cheat Sheet

The big picture

AI ⊃ ML ⊃ Deep Learning

Types of ML at a glance

Type	Data	Goal	Example
Supervised	Labelled	Predict labels	Spam detection, house prices
Unsupervised	Unlabelled	Find hidden structure	Customer clustering
Semi-supervised	Few labelled + many unlabelled	Learn with less labelling	Photo tagging
Self-supervised	Unlabelled (self-generated labels)	Learn representations	Predicting masked words
Reinforcement	Interaction + rewards	Maximise cumulative reward	Game-playing agents

Other ways to categorise

Dimension	Option A	Option B
When it learns	Batch (all at once, offline)	Online (incrementally, continuously)
How it generalises	Instance-based (compare to stored examples)	Model-based (learn a function)

Common problems

Problem	Fix
Not enough data	Collect more, use semi/self-supervised approaches
Bad quality data	Clean, preprocess, fix labels
Overfitting	More data, simpler model, regularisation
Underfitting	More powerful model, better features
Model goes stale	Monitor and retrain

ML lifecycle in one line

Define → Collect → Prepare → Engineer Features → Train → Evaluate → Tune → Deploy → Monitor (and loop!)

---

✅ Final Takeaways

1. ML = learning patterns from data instead of hand-writing rules.
2. Choose the learning type based on the data and labels you have.
3. Data quality matters more than almost anything else.
4. Always evaluate on unseen data to check true generalisation.
5. ML is an ongoing cycle, not a one-time build. Deploy, monitor, improve, repeat.

---

Friendly, beginner-to-advanced notes on the math that powers Machine Learning. Every concept comes with a plain-English idea, a small worked example, and a note on why ML cares.

---

📑 Table of Contents

1. Linear Algebra
2. Calculus
3. Probability
4. Statistics
5. Optimization
6. Information Theory
7. Quick Revision Cheat Sheet

🗺️ How the pieces fit together

 Linear Algebra ──► how data and models are REPRESENTED (vectors, matrices)
 Calculus       ──► how models CHANGE when we tweak their parameters (gradients)
 Probability    ──► how we handle UNCERTAINTY (distributions, Bayes)
 Statistics     ──► how we LEARN FROM DATA and judge results (estimates, tests)
 Optimization   ──► how we FIND the best parameters (gradient descent)
 Information Th.──► how we MEASURE uncertainty and difference (entropy, KL, loss)

---

1. Linear Algebra

The big idea: Linear algebra is the language of data. A dataset is a matrix, an image is a grid of numbers, a model's weights are vectors and matrices. Almost every ML computation is a linear algebra operation under the hood.

1.1 Scalars, Vectors, Matrices, Tensors

These are the four "containers" for numbers, differing only in how many dimensions they have.

Object	Dimensions	Notation (usual)	Example	ML example
Scalar	0-D (a single number)	x (lowercase italic)	7	A learning rate, a bias value
Vector	1-D (a list of numbers)	v (bold lowercase)	[2, 5, 1]	One data sample with 3 features
Matrix	2-D (a table of numbers)	A (bold uppercase)	[[1,2],[3,4]]	A dataset: rows = samples, columns = features
Tensor	n-D (a general array)	T	A 3-D grid	A colour image (height × width × 3 channels), a batch of images (4-D)

Scalar     Vector        Matrix           Tensor (3-D)
  5        [1]           [1 2 3]          a stack of matrices,
           [4]           [4 5 6]          like a cube of numbers
           [2]

Shape tells you the size of each dimension. A matrix with 100 rows and 5 columns has shape (100, 5).

💡 Think of it as a ladder: scalar → vector (list of scalars) → matrix (list of vectors) → tensor (list of matrices).

1.2 Matrix Operations

Addition and subtraction

Done element by element. Both matrices must have the same shape.

[1 2]   [5 6]   [ 6  8]
[3 4] + [7 8] = [10 12]

Scalar multiplication

Multiply every element by the number.

  [1 2]   [3  6]
3×[3 4] = [9 12]

Matrix multiplication

This is not element by element. The entry in row i, column j of the result is the dot product of row i of A with column j of B.

A = [1 2]    B = [5 6]
    [3 4]        [7 8]

A×B = [1·5+2·7   1·6+2·8]   [19 22]
      [3·5+4·7   3·6+4·8] = [43 50]

Rules to remember

• Shape rule: (m × n) · (n × p) = (m × p). The inner dimensions must match.
• Not commutative: in general, A·B ≠ B·A.
• Associative: (A·B)·C = A·(B·C).
• Distributive: A·(B + C) = A·B + A·C.

Element-wise (Hadamard) product

Multiply matching positions only (written A ⊙ B). Shapes must be the same. This shows up constantly in neural networks.

🧠 Why ML cares

A whole layer of a neural network is essentially output = W·x + b, a matrix multiplication plus a bias. Predicting for 1,000 samples at once is one big matrix multiplication.

1.3 Transpose

Flip a matrix over its diagonal: rows become columns and columns become rows. Written Aᵀ.

A = [1 2 3]        Aᵀ = [1 4]
    [4 5 6]             [2 5]
                        [3 6]

(2×3) becomes (3×2).

Properties

• (Aᵀ)ᵀ = A
• (A + B)ᵀ = Aᵀ + Bᵀ
• (A·B)ᵀ = Bᵀ·Aᵀ (note the order reverses!)
• A matrix is symmetric if A = Aᵀ. Covariance matrices are always symmetric.

1.4 Identity Matrix and Inverse

Identity matrix (I)

The matrix equivalent of the number 1. It has 1s on the diagonal and 0s elsewhere. Multiplying by it changes nothing: A·I = I·A = A.

I₃ = [1 0 0]
     [0 1 0]
     [0 0 1]

Inverse matrix (A⁻¹)

The matrix equivalent of "1 divided by a number". It undoes what A does:

A · A⁻¹ = A⁻¹ · A = I

Key facts

• Only square matrices can have an inverse, and not all of them do.
• A matrix with an inverse is called invertible (or non-singular). One without is singular.
• A matrix is invertible if and only if its determinant ≠ 0.
• (A·B)⁻¹ = B⁻¹·A⁻¹

2×2 shortcut: For A = [[a, b], [c, d]]:

A⁻¹ = 1/(ad − bc) × [ d  −b]
                    [−c   a]

Example: A = [[4, 2], [1, 3]] has ad − bc = 12 − 2 = 10, so

A⁻¹ = (1/10) × [ 3 −2]   =  [ 0.3 −0.2]
               [−1  4]      [−0.1  0.4]

🧠 Why ML cares

Solving linear systems like A·x = b is x = A⁻¹·b. Linear Regression's closed-form solution (the "normal equation") uses an inverse. In practice, computers use more stable numerical methods rather than computing the inverse directly.

1.5 Determinant

A single number computed from a square matrix that tells you how the matrix scales space.

2×2: det [[a, b], [c, d]] = ad − bc

3×3 (expansion along the first row):

det(A) = a(ei − fh) − b(di − fg) + c(dh − eg)     for A = [[a,b,c],[d,e,f],[g,h,i]]

What it means

• |det| = the factor by which areas (2-D) or volumes (3-D) get scaled by the matrix.
• Sign: a negative determinant means the transformation flips orientation (like a mirror).
• det = 0 means the matrix squashes space into a lower dimension (a plane into a line). It loses information, so it is not invertible.

Properties

• det(A·B) = det(A) · det(B)
• det(Aᵀ) = det(A)
• det(A⁻¹) = 1 / det(A)
• det(I) = 1
• The determinant equals the product of the eigenvalues.

1.6 Rank

Rank = the number of linearly independent rows (or columns) in a matrix. In other words, how much genuinely distinct information the matrix holds.

[1 2 3]
[2 4 6]   ← this row is just 2 × row 1 (redundant!)
[1 0 1]

Rank = 2  (only two independent rows)

Key facts

• Rank ≤ min(rows, columns).
• Full rank means rank equals the smaller dimension (no redundancy).
• A square n×n matrix is invertible only if its rank is n.
• Rank-deficient matrices have redundant (dependent) information.

🧠 Why ML cares

If two features are perfectly redundant (one is a multiple or combination of others), the feature matrix is rank-deficient, which can break some algorithms. Rank also tells us the "true dimensionality" of data.

1.7 Dot Product

For two vectors of the same length, multiply matching entries and add them up. The result is a scalar.

a = [1, 2, 3]    b = [4, 5, 6]
a · b = 1×4 + 2×5 + 3×6 = 4 + 10 + 18 = 32

Geometric meaning

a · b = ‖a‖ · ‖b‖ · cos(θ)

where ‖a‖ is the length (norm) of a, ‖a‖ = √(a₁² + a₂² + …), and θ is the angle between the vectors.

Dot product value	What it says
Positive	Vectors point in a similar direction (angle < 90°)
Zero	Vectors are orthogonal (perpendicular, 90°)
Negative	Vectors point in opposite-ish directions (angle > 90°)

Useful by-products

• Cosine similarity = (a · b) / (‖a‖ ‖b‖), ranging from −1 to 1. It measures how aligned two vectors are regardless of their length.
• Vector length: ‖a‖ = √(a · a)
• Projection of a onto b = ((a · b) / (b · b)) × b, the "shadow" of a on b.

🧠 Why ML cares

A neuron computes w · x + b, a dot product between weights and inputs. Similarity between documents, embeddings, and recommendations is often a dot product or cosine similarity.

1.8 Vector Spaces

A vector space is a collection of vectors where you can add any two vectors and scale any vector by a number, and the result stays inside the collection. (Think of the 2-D plane ℝ² or 3-D space ℝ³.)

Core building blocks

Concept	Meaning	Example
Linear combination	A sum of scaled vectors: c₁v₁ + c₂v₂ + …	2·[1,0] + 3·[0,1] = [2,3]
Span	All vectors you can build from linear combinations of a set	The span of [1,0] and [0,1] is the whole plane
Linear independence	No vector in the set can be built from the others	[1,0] and [0,1] are independent; [1,2] and [2,4] are not
Basis	A set of linearly independent vectors that spans the space	[1,0] and [0,1] (the standard basis)
Dimension	The number of vectors in a basis	ℝ² has dimension 2
Subspace	A smaller vector space living inside a bigger one	A line through the origin inside 3-D space

Related ideas

• Column space of a matrix = the span of its columns. Its dimension equals the rank.
• Null space = all vectors x such that A·x = 0.
• Orthogonal vectors have zero dot product. An orthonormal basis has vectors that are perpendicular and have length 1, which makes computations clean.

🧠 Why ML cares

Features live in a vector space. Techniques like dimensionality reduction find a smaller subspace that captures most of the information in the data.

1.9 Eigenvalues and Eigenvectors

A matrix usually rotates and stretches vectors. But some special vectors only get stretched or shrunk and keep their direction. These are eigenvectors, and the stretch factor is the eigenvalue.

A · v = λ · v

• v = eigenvector (non-zero)
• λ (lambda) = eigenvalue

How to find them (for a square matrix)

1. Solve the characteristic equation: det(A − λI) = 0 to find the eigenvalues λ.
2. For each λ, solve (A − λI)·v = 0 to find the eigenvector v.

Worked example

A = [[4, 1], [2, 3]]

1. det(A − λI) = (4−λ)(3−λ) − 2 = λ² − 7λ + 10 = 0 gives λ = 5 and λ = 2.
2. For λ = 5: (A − 5I)v = 0 gives v = [1, 1]. Check: A·[1,1] = [5,5] = 5·[1,1] ✅
3. For λ = 2: gives v = [1, −2]. Check: A·[1,−2] = [2,−4] = 2·[1,−2] ✅

Handy facts

• Sum of eigenvalues = trace (sum of the diagonal). Here 5 + 2 = 7 = 4 + 3 ✅
• Product of eigenvalues = determinant. Here 5 × 2 = 10 = det(A) ✅
• A matrix is singular (non-invertible) if any eigenvalue is 0.
• Symmetric matrices have real eigenvalues and orthogonal eigenvectors.

Eigendecomposition

A diagonalizable matrix can be written as:

A = V · Λ · V⁻¹

where the columns of V are eigenvectors and Λ is a diagonal matrix of eigenvalues. For a symmetric matrix this becomes A = Q·Λ·Qᵀ.

🧠 Why ML cares

The eigenvectors of a data covariance matrix point along the directions of greatest variation, and the eigenvalues say how much variation lies along each. This is the heart of Principal Component Analysis (PCA).

1.10 Singular Value Decomposition (SVD)

Eigendecomposition only works for certain square matrices. SVD works for any matrix (any shape), which makes it extremely general and powerful.

A = U · Σ · Vᵀ

Piece	Shape (for A of size m × n)	Meaning
U	m × m	Orthogonal matrix of left singular vectors
Σ (Sigma)	m × n	Diagonal matrix of singular values σ₁ ≥ σ₂ ≥ … ≥ 0
Vᵀ	n × n	Orthogonal matrix of right singular vectors (transposed)

Intuition: Any linear transformation can be broken into three simple steps: rotate (Vᵀ), stretch along the axes (Σ), then rotate again (U).

Connection to eigenvalues: The singular values of A are the square roots of the eigenvalues of AᵀA.

Low-rank approximation

Keep only the largest k singular values (set the rest to zero). You get the best rank-k approximation of A:

A ≈ Uₖ · Σₖ · Vₖᵀ

This stores far fewer numbers while keeping the most important structure.

🧠 Why ML cares

• Dimensionality reduction (PCA can be computed via SVD)
• Data and image compression
• Noise reduction (small singular values often correspond to noise)
• Recommender systems (matrix factorization)
• Handling rank-deficient or ill-conditioned matrices

---

2. Calculus

The big idea: Calculus tells us how things change. In ML, we constantly ask: "If I nudge this parameter slightly, how much does my error change?" The answer is a derivative, and it tells us which way to adjust the model.

2.1 Limits

A limit describes the value a function approaches as the input gets closer and closer to some point, even if it never actually reaches it.

lim   f(x) = L
x→a

"As x gets close to a, f(x) gets close to L."

Example: lim(x→2) (x² − 4)/(x − 2). Plugging in x = 2 gives 0/0, which is undefined. But (x²−4)/(x−2) = (x+2), so the limit is 4.

🧠 Why it matters

Limits are the foundation for the derivative. Without them, "instantaneous rate of change" has no precise meaning. Limits as x → ∞ also describe the long-term behaviour of functions (e.g., a sigmoid flattening toward 1).

2.2 Derivatives

The derivative of f at a point is the instantaneous rate of change, the slope of the tangent line at that point.

f′(x) = lim   [ f(x + h) − f(x) ] / h
        h→0

Reading the derivative

Sign of f′(x)	Meaning
Positive	f is increasing at x
Negative	f is decreasing at x
Zero	f is flat at x, a possible minimum, maximum, or saddle point

Common derivative rules

Rule	Formula
Constant	d/dx (c) = 0
Power	d/dx (xⁿ) = n·xⁿ⁻¹
Constant multiple	d/dx (c·f) = c·f′
Sum	d/dx (f + g) = f′ + g′
Product	d/dx (f·g) = f′·g + f·g′
Quotient	d/dx (f/g) = (f′·g − f·g′) / g²
Exponential	d/dx (eˣ) = eˣ
Natural log	d/dx (ln x) = 1/x
Sine / Cosine	d/dx (sin x) = cos x, d/dx (cos x) = −sin x

Example: f(x) = 3x² + 2x gives f′(x) = 6x + 2. At x = 1, the slope is 8.

Derivatives of functions common in ML

• Sigmoid σ(x) = 1/(1+e⁻ˣ) has derivative σ(x)·(1 − σ(x))
• tanh has derivative 1 − tanh²(x)
• ReLU max(0, x) has derivative 0 for x < 0 and 1 for x > 0

2.3 Partial Derivatives

Most ML functions depend on many variables (many parameters). A partial derivative measures the change in the function when you vary one variable while holding all others constant.

Written ∂f/∂x.

Example: f(x, y) = x²y + 3y

• ∂f/∂x = 2xy (treat y as a constant)
• ∂f/∂y = x² + 3 (treat x as a constant)

🧠 Why ML cares

A model has thousands or millions of parameters. A partial derivative tells us how the loss changes with respect to each one individually.

2.4 Chain Rule

The chain rule handles functions inside functions (composition). If y = f(g(x)):

dy/dx = f′(g(x)) · g′(x)

Or, with u = g(x): dy/dx = (dy/du) · (du/dx). "Derivative of the outside × derivative of the inside."

Example: y = (3x + 1)². Let u = 3x + 1, so y = u². dy/dx = 2u · 3 = 6(3x + 1).

Multivariable chain rule

If z = f(x, y) where x = x(t) and y = y(t):

dz/dt = (∂f/∂x)(dx/dt) + (∂f/∂y)(dy/dt)

🧠 Why ML cares

A neural network is a long chain of functions (layer after layer). Backpropagation is simply the chain rule applied repeatedly, passing error derivatives backward from the output through each layer to find how every weight affects the loss.

2.5 Gradients

For a function of many variables, the gradient collects all the partial derivatives into one vector:

            [ ∂f/∂x₁ ]
∇f(x)  =    [ ∂f/∂x₂ ]
            [   ⋮    ]
            [ ∂f/∂xₙ ]

Two key properties

1. The gradient points in the direction of steepest increase of f.
2. The negative gradient (−∇f) points in the direction of steepest decrease.

Example: f(x, y) = x² + 3y gives ∇f = [2x, 3]. At the point (1, 5), the gradient is [2, 3].

Magnitude ‖∇f‖ tells you how steep the slope is. At a minimum, ∇f = 0.

   Loss
    │\            /
    │ \          /
    │  \        /     ← gradient points uphill,
    │   \__  __/         we step the OPPOSITE way (downhill)
    │      \/
    └────────────── parameter

2.6 Jacobian

When a function has many inputs and many outputs (a vector-valued function f: ℝⁿ → ℝᵐ), the Jacobian is the matrix of all first-order partial derivatives:

        [ ∂f₁/∂x₁  ∂f₁/∂x₂  …  ∂f₁/∂xₙ ]
J  =    [ ∂f₂/∂x₁  ∂f₂/∂x₂  …  ∂f₂/∂xₙ ]       (shape m × n)
        [    ⋮         ⋮           ⋮    ]
        [ ∂fₘ/∂x₁  ∂fₘ/∂x₂  …  ∂fₘ/∂xₙ ]

Each row is the gradient of one output function.

Example: f(x, y) = [x²y, 5x + sin y]

J = [ 2xy    x²   ]
    [  5    cos y ]

Notes

• For a scalar function (m = 1), the Jacobian is just the gradient written as a row.
• The Jacobian is the best linear approximation of f near a point.
• The chain rule in matrix form: Jacobian of a composition = product of Jacobians.

🧠 Why ML cares

Layers of neural networks map vectors to vectors. Their Jacobians describe how each output changes with each input, and are what backpropagation effectively multiplies together.

2.7 Hessian

The Hessian is the matrix of all second-order partial derivatives of a scalar function. It describes the curvature of the function.

        [ ∂²f/∂x₁²      ∂²f/∂x₁∂x₂  …  ∂²f/∂x₁∂xₙ ]
H  =    [ ∂²f/∂x₂∂x₁    ∂²f/∂x₂²    …  ∂²f/∂x₂∂xₙ ]       (shape n × n)
        [      ⋮              ⋮               ⋮       ]
        [ ∂²f/∂xₙ∂x₁    ∂²f/∂xₙ∂x₂  …  ∂²f/∂xₙ²    ]

(The Hessian is the Jacobian of the gradient.) It is symmetric when the second derivatives are continuous.

Example: f(x, y) = x² + xy + y²

∇f = [2x + y,  x + 2y]       H = [2 1]
                                 [1 2]

Using the Hessian to classify a critical point (where ∇f = 0)

Hessian's eigenvalues	Hessian type	The point is a…
All positive	Positive definite	Local minimum (bowl-shaped)
All negative	Negative definite	Local maximum (hill-shaped)
Mixed signs	Indefinite	Saddle point
Some zero	Semi-definite	Test inconclusive

For the example above, the eigenvalues of H are 3 and 1, both positive, so f is convex (bowl-shaped).

🧠 Why ML cares

Curvature tells us how quickly the slope itself changes, which guides step sizes. Second-order optimization methods (like Newton's method) use the Hessian, though it is expensive for models with millions of parameters.

2.8 Gradient-Based Optimization (Preview)

Putting calculus to work: to minimize a function, repeatedly move opposite to the gradient.

new parameters = old parameters − (step size) × gradient

This is the core idea of gradient descent, covered in full in Section 5. Calculus supplies the direction (the gradient), and optimization supplies the strategy for using it.

---

3. Probability

The big idea: The real world is noisy and uncertain. Probability gives us a precise language to describe uncertainty, so models can say "I'm 87% sure this is a cat" rather than just "cat."

3.1 Basic Building Blocks

Term	Meaning	Example (rolling a die)
Experiment	A process with an uncertain outcome	Rolling the die
Sample space (Ω or S)	The set of all possible outcomes	{1, 2, 3, 4, 5, 6}
Outcome	One single result	4
Event	A subset of the sample space	"Even number" = {2, 4, 6}
Probability P(A)	A number from 0 to 1 measuring how likely event A is	P(even) = 3/6 = 0.5

The three axioms (rules) of probability

1. P(A) ≥ 0 for any event A (never negative)
2. P(Ω) = 1 (something must happen)
3. For mutually exclusive events, P(A ∪ B) = P(A) + P(B)

Handy rules

• Complement: P(not A) = 1 − P(A)
• Addition rule (general): P(A or B) = P(A) + P(B) − P(A and B)
• Equally likely outcomes: P(A) = (favourable outcomes) / (total outcomes)

3.2 Random Variables

A random variable (RV) is a variable whose value is a numerical outcome of a random process. It's a rule that attaches a number to each outcome.

Example: Flip 3 coins. Let X = number of heads. X can be 0, 1, 2 or 3.

Two types

	Discrete RV	Continuous RV
Values	Countable (0, 1, 2…)	Any value in a range (height, time)
Described by	PMF (Probability Mass Function)	PDF (Probability Density Function)
Probability of an exact value	P(X = x) can be > 0	P(X = x) = 0 for any single exact value
Probability over a range	Sum the PMF	Area under the PDF curve
Total	Sum of all probabilities = 1	Total area under the curve = 1

CDF (Cumulative Distribution Function)

Works for both types: F(x) = P(X ≤ x), the probability the variable is at most x. It rises from 0 to 1.

⚠️ A PDF value is a density, not a probability, so it can exceed 1. Only the area under it is a probability.

3.3 Conditional Probability

The probability of event A given that event B has already happened:

P(A | B) = P(A and B) / P(B)       (requires P(B) > 0)

Knowing B shrinks the sample space to just B, and we ask what fraction of B is also A.

Example: Draw one card from a standard deck. Given the card is a face card (J, Q, K: 12 cards), what's the chance it's a King? P(King | Face) = 4/12 = 1/3.

Multiplication rule (chain rule of probability)

P(A and B) = P(A | B) · P(B) = P(B | A) · P(A)

For many events: P(A,B,C) = P(A) · P(B|A) · P(C|A,B)

Law of total probability

If B₁, B₂, … Bₙ split the sample space into non-overlapping pieces:

P(A) = Σ P(A | Bᵢ) · P(Bᵢ)

3.4 Bayes' Theorem

Lets us flip a conditional probability, going from P(B | A) to P(A | B):

              P(B | A) · P(A)
P(A | B)  =  ─────────────────
                   P(B)

In "learning" language:

Posterior  =  (Likelihood × Prior) / Evidence

P(hypothesis | data)  =  P(data | hypothesis) · P(hypothesis) / P(data)

Term	Meaning
Prior P(A)	What you believed before seeing the data
Likelihood P(B|A)	How probable the data is if the hypothesis is true
Evidence P(B)	Overall probability of the data (a normalising factor)
Posterior P(A|B)	Your updated belief after seeing the data

Worked example: the medical test

• 1% of people have a disease: P(D) = 0.01
• Test is positive for 90% of sick people: P(+|D) = 0.90
• Test is wrongly positive for 5% of healthy people: P(+|¬D) = 0.05

Evidence: P(+) = 0.90 × 0.01 + 0.05 × 0.99 = 0.009 + 0.0495 = 0.0585

Posterior: P(D|+) = (0.90 × 0.01) / 0.0585 ≈ 0.154

So even after a positive test, there is only about a 15.4% chance of actually having the disease. The disease is so rare that false positives outnumber true positives. Ignoring the prior (the base rate) is a classic mistake.

🧠 Why ML cares

Bayes' theorem underlies Naive Bayes classifiers, Bayesian inference, and the idea of updating beliefs as more data arrives.

3.5 Independence

Two events are independent if knowing one tells you nothing about the other.

Independent  ⟺  P(A and B) = P(A) · P(B)
            ⟺  P(A | B) = P(A)

Example: Two separate coin flips are independent. P(both heads) = 0.5 × 0.5 = 0.25.

Related ideas

• Mutually exclusive ≠ independent. Mutually exclusive events can't happen together. If one happens, the other definitely doesn't, so they're actually strongly dependent.
• Conditional independence: A and B are independent given C if P(A, B | C) = P(A|C) · P(B|C). Naive Bayes assumes features are conditionally independent given the class.
• i.i.d. ("independent and identically distributed"): the usual assumption that data samples are drawn independently from the same distribution.

3.6 Expectation (Expected Value)

The long-run average value of a random variable, a probability-weighted average of its possible values.

Discrete:    E[X] = Σ x · P(X = x)
Continuous:  E[X] = ∫ x · f(x) dx

Example (fair die): E[X] = (1+2+3+4+5+6)/6 = 3.5. (You never roll a 3.5, but it's the average over many rolls.)

Properties

• Linearity: E[aX + b] = a·E[X] + b
• E[X + Y] = E[X] + E[Y] (always true, even if X and Y are dependent)
• If X and Y are independent: E[X·Y] = E[X]·E[Y]
• For a function: E[g(X)] = Σ g(x) · P(X = x)

3.7 Variance and Standard Deviation

Variance measures how spread out a random variable is around its mean.

Var(X) = E[(X − μ)²]  =  E[X²] − (E[X])²

Standard deviation: σ = √Var(X), in the same units as X.

Example (fair die): E[X²] = (1+4+9+16+25+36)/6 = 91/6 ≈ 15.17, and (E[X])² = 12.25, so Var = 15.17 − 12.25 ≈ 2.92 and σ ≈ 1.71.

Properties

• Var(aX + b) = a² · Var(X) (adding a constant doesn't change spread)
• If X and Y are independent: Var(X + Y) = Var(X) + Var(Y)

3.8 Covariance

Measures how two random variables vary together:

Cov(X, Y) = E[(X − μₓ)(Y − μᵧ)] = E[XY] − E[X]·E[Y]

Cov(X, Y)	Interpretation
> 0	When X is above its mean, Y tends to be above its mean too
< 0	When X is above its mean, Y tends to be below its mean
≈ 0	No linear relationship

Notes

• Cov(X, X) = Var(X)
• Independent variables have Cov = 0, but zero covariance doesn't guarantee independence (it only rules out a linear relationship).
• The size of covariance depends on units, so it's hard to compare. That's why we use correlation (see Statistics).

Covariance matrix

For a vector of d variables, the covariance matrix Σ is a d × d matrix whose entry (i, j) is Cov(Xᵢ, Xⱼ). The diagonal holds the variances. It is symmetric and positive semi-definite.

3.9 Probability Distributions

A probability distribution describes how probability is spread over the possible values of a random variable.

Discrete distributions

Distribution	Describes	Parameters	Mean / Variance	Example
Bernoulli	One yes/no trial	p (success prob.)	p / p(1−p)	A single coin flip, one click/no-click
Binomial	Number of successes in n independent Bernoulli trials	n, p	np / np(1−p)	Heads in 10 flips
Categorical	One trial with k possible outcomes	probabilities p₁…pₖ	,	A single die roll, class prediction
Poisson	Count of events in a fixed interval	λ (average rate)	λ / λ	Emails received per hour

Continuous distributions

Distribution	Describes	Parameters	Mean / Variance
Uniform	All values in [a, b] equally likely	a, b	(a+b)/2 / (b−a)²/12
Gaussian (Normal)	The famous bell curve	μ (mean), σ² (variance)	μ / σ²
Exponential	Waiting time between random events	λ (rate)	1/λ / 1/λ²

The Gaussian (Normal) distribution: the star of ML

f(x) = 1 / (σ√(2π)) · exp( −(x − μ)² / (2σ²) )

Written X ~ N(μ, σ²).

          .-'''-.
        .'       '.        ← peak at the mean μ
      .'           '.
   _.'               '._   ← spread controlled by σ
  ───────────────────────
  μ−2σ   μ−σ   μ   μ+σ  μ+2σ

The 68–95–99.7 rule

• ≈ 68% of values fall within 1σ of the mean
• ≈ 95% within 2σ
• ≈ 99.7% within 3σ

Standard normal: N(0, 1). Any normal variable can be converted to it by the z-score: z = (x − μ) / σ.

Why it's everywhere: The Central Limit Theorem says the average of many independent random variables tends toward a normal distribution, regardless of the original distribution. Many natural measurements, and many noise assumptions in ML, are therefore treated as Gaussian.

Multivariate Gaussian: The extension to many variables, described by a mean vector μ and a covariance matrix Σ.

---

4. Statistics

The big idea: Probability goes from a known model to data. Statistics goes the other way: from observed data back to conclusions about the underlying world. It's how we summarise data, estimate things, and decide whether results are real or just luck.

4.1 Descriptive Statistics: Summarising Data

We'll use this tiny dataset throughout: 2, 3, 3, 5, 7

Measures of central tendency ("the typical value")

Measure	Definition	Our data
Mean (average)	Sum ÷ count: x̄ = (Σxᵢ) / n	(2+3+3+5+7)/5 = 4
Median	The middle value when sorted (average the two middle values if the count is even)	3
Mode	The most frequent value	3

When to use which

• Mean: uses every value, but is sensitive to outliers.
• Median: robust to outliers. Better for skewed data like incomes or house prices.
• Mode: the only one that works for categorical data (e.g., most common colour).

Example: for 2, 3, 3, 5, 100, the mean jumps to 22.6, while the median stays at 3.

In a symmetric distribution, mean ≈ median ≈ mode. In a right-skewed one, mean > median > mode.

Measures of spread ("how scattered is the data")

Measure	Formula	Our data
Range	max − min	7 − 2 = 5
Variance	Average squared distance from the mean	see below
Standard deviation	√variance	see below

Population vs sample formulas

Population variance:  σ² = Σ(xᵢ − μ)² / N
Sample variance:      s² = Σ(xᵢ − x̄)² / (n − 1)

With data 2,3,3,5,7 and mean 4, the squared deviations are 4, 1, 1, 1, 9 (sum = 16):

• Population: σ² = 16/5 = 3.2, σ ≈ 1.79
• Sample: s² = 16/4 = 4, s = 2

Why n − 1 for samples? (Bessel's correction): A sample's values tend to sit closer to the sample mean than to the true mean, so dividing by n would underestimate the true variance. Dividing by n − 1 corrects this bias.

Standard deviation is easier to interpret than variance because it's in the same units as the data.

Other spread measures: Interquartile range (IQR) = Q3 − Q1 (the spread of the middle 50%), useful for spotting outliers.

4.2 Covariance and Correlation

Sample covariance:

Cov(X, Y) = Σ (xᵢ − x̄)(yᵢ − ȳ) / (n − 1)

(Interpretation is the same as in Section 3.8.)

Correlation (Pearson's r)

Covariance rescaled to be unit-free and bounded between −1 and +1:

r = Cov(X, Y) / (σₓ · σᵧ)

r value	Meaning
+1	Perfect positive linear relationship
0	No linear relationship
−1	Perfect negative linear relationship
≈ ±0.7 or more	Commonly considered a strong linear relationship

Important cautions

• ⚠️ Correlation ≠ causation. Ice-cream sales and drowning both rise in summer, but one doesn't cause the other (a hidden third factor, the weather, drives both).
• Pearson's r only captures linear relationships. A perfect curve (like y = x²) can have r ≈ 0.
• It's sensitive to outliers.
• A correlation matrix shows pairwise correlations among many variables. It helps spot redundant features.

4.3 Population vs Sample

	Population	Sample
Definition	The entire group you care about	A subset drawn from the population
Size	Often huge or impossible to fully measure	Manageable
Numbers describing it are called	Parameters (μ, σ)	Statistics (x̄, s)
Example	All users of an app	1,000 randomly chosen users

We almost never observe the whole population, so we use sample statistics to estimate population parameters.

Sampling methods

Method	How it works
Simple random	Every member has an equal chance of being picked
Stratified	Split population into groups (strata) and sample from each
Systematic	Pick every k-th member from a list
Cluster	Randomly choose whole groups and include everyone in them
Convenience	Pick whoever is easy to reach (often biased)

Sampling bias: if the sample doesn't represent the population (e.g., surveying only people who answer phone calls), conclusions will be skewed. A larger, truly random sample reduces error.

Sampling distribution and the Central Limit Theorem

If you take many samples and compute each one's mean, those means form a sampling distribution.

• Its mean equals the population mean μ.
• Its spread is the standard error: SE = σ / √n. It shrinks as sample size n grows.
• By the Central Limit Theorem, for large n this distribution is approximately normal, even if the population isn't.

4.4 Hypothesis Testing

A formal procedure for deciding whether the evidence in a sample is strong enough to reject a default assumption. Think of it like a courtroom: innocent until proven guilty beyond reasonable doubt.

The ingredients

Term	Meaning
Null hypothesis (H₀)	The default claim: "no effect / no difference"
Alternative hypothesis (H₁ or Hₐ)	What you suspect instead: "there is an effect / difference"
Test statistic	A number computed from the data that measures how far it is from what H₀ predicts (z, t, χ², F…)
Significance level (α)	The threshold for "surprising enough" (commonly 0.05)
p-value	See Section 4.6
Decision	Reject H₀ or fail to reject H₀

The step-by-step process

1. State H₀ and H₁.
2. Choose the significance level α (e.g., 0.05).
3. Collect data and compute the test statistic.
4. Compute the p-value (or compare the statistic to a critical value).
5. Decide: if p ≤ α, reject H₀. Otherwise, fail to reject H₀.

Note the careful wording: we say "fail to reject" H₀, not "accept" it. Lack of evidence isn't proof of no effect.

One-tailed vs two-tailed tests

• Two-tailed: H₁ says "different" (≠). Looks for extreme values in both directions.
• One-tailed: H₁ says "greater than" or "less than". Looks in one direction only.

Types of errors

	H₀ is actually true	H₀ is actually false
Reject H₀	❌ Type I error (false positive), probability = α	✅ Correct (probability = power, 1 − β)
Fail to reject H₀	✅ Correct	❌ Type II error (false negative), probability = β

• Power (1 − β): the chance of correctly detecting a real effect. It increases with larger sample sizes and larger true effects.
• Lowering α reduces Type I errors but tends to increase Type II errors.

Common tests (names to recognise)

• z-test / t-test: compare means (t-test when σ is unknown, especially for small samples)
• Chi-square test: relationships between categorical variables
• ANOVA (F-test): compare means across 3+ groups

Quick example (z-test)

A website's average load time was historically μ₀ = 3.0 s with known σ = 0.6 s. After an update, 36 pages averaged x̄ = 2.8 s. Did it truly improve?

• H₀: μ = 3.0 vs H₁: μ < 3.0 (one-tailed)
• z = (x̄ − μ₀) / (σ/√n) = (2.8 − 3.0) / (0.6/6) = −0.2 / 0.1 = −2.0
• For z = −2.0, the one-tailed p-value ≈ 0.023, which is below 0.05, so reject H₀. There is statistically significant evidence of improvement.

4.5 Confidence Intervals

A confidence interval (CI) gives a range of plausible values for a population parameter, instead of a single-point guess.

CI = point estimate ± margin of error
   = x̄ ± (critical value) × (standard error)

For a mean with known σ:

x̄ ± z* · σ/√n

Confidence level	z* (critical value)
90%	1.645
95%	1.96
99%	2.576

(If σ is unknown, which is typical, use the t-distribution with the sample's s instead.)

Example: Sample of n = 100, x̄ = 50, σ = 10. SE = 10/√100 = 1, so 95% CI = 50 ± 1.96 × 1 = (48.04, 51.96).

How to interpret it correctly

✅ "If we repeated this sampling procedure many times, about 95% of the intervals built this way would contain the true population mean."

❌ It does not mean "there's a 95% probability that the true mean is in this particular interval." (The true mean is fixed. The interval is what varies.)

What changes the width?

• Larger sample size gives a narrower interval (more precision)
• Higher confidence level gives a wider interval
• More variability in the data gives a wider interval

4.6 P-Values

The p-value is the probability of getting results at least as extreme as what you observed, assuming the null hypothesis is true.

• Small p-value (≤ α, typically 0.05): the data would be very surprising if H₀ were true, so we have evidence against H₀.
• Large p-value: the data is consistent with H₀, so there's not enough evidence to reject it.

p-value	Informal reading
< 0.01	Strong evidence against H₀
0.01 – 0.05	Moderate evidence against H₀
> 0.05	Little or no evidence against H₀

⚠️ What a p-value is NOT

• ❌ It is not the probability that H₀ is true.
• ❌ It is not the probability that the result happened "by chance" in a simple sense.
• ❌ It does not measure the size or importance of an effect. With huge samples, tiny, meaningless differences can be "significant."
• ❌ p = 0.05 isn't a magic line. It's a convention.

Connection to confidence intervals

They tell a consistent story: a two-sided test at α = 0.05 rejects H₀ exactly when the 95% CI doesn't contain the null value.

Multiple testing warning

If you run many tests, some will come out "significant" by pure luck (about 1 in 20 at α = 0.05). This is called p-hacking when abused, so corrections (e.g., Bonferroni) exist.

---

5. Optimization

The big idea: Training a model means finding the best parameters, the ones that make the model's mistakes as small as possible. That search is an optimization problem.

5.1 Objective Functions

An objective function is the quantity we want to minimize or maximize.

minimize   f(θ)        (θ = the parameters we can adjust)

• When we minimize it, it's also called a cost or loss function.
• When we maximize it (e.g., likelihood, reward), it's called a utility or score.
• Maximizing f is the same as minimizing −f, so we usually just talk about minimization.

Optimal point: θ* = argmin f(θ), the parameter values that give the lowest value.

Local vs global minimum

   f(θ)
    │  \         /\
    │   \  /\   /  \        ← local minima (valleys that aren't the lowest)
    │    \/  \_/    \___
    │         ↑ global minimum (the lowest point of all)
    └──────────────────── θ

• Global minimum: the very lowest value across the whole domain.
• Local minimum: lowest in its neighbourhood only.
• Saddle point: gradient is zero but it's a minimum in some directions and a maximum in others.

5.2 Loss Functions

A loss function L(y, ŷ) measures how wrong a single prediction is (ŷ = prediction, y = true value). The cost function is typically the average loss over the whole dataset.

For regression (predicting numbers)

Loss	Formula	Behaviour
MSE (Mean Squared Error)	(1/n) Σ (yᵢ − ŷᵢ)²	Heavily penalises large errors. Smooth and easy to differentiate
MAE (Mean Absolute Error)	(1/n) Σ |yᵢ − ŷᵢ|	Robust to outliers. Not smooth at 0
RMSE	√MSE	Same units as the target
Huber loss	Quadratic for small errors, linear for large	A blend of MSE and MAE

For classification (predicting categories)

Loss	Formula / Idea	Notes
Binary cross-entropy (log loss)	−[y·log(ŷ) + (1−y)·log(1−ŷ)]	For yes/no problems with probability outputs
Categorical cross-entropy	−Σ yₖ · log(ŷₖ)	For multi-class problems
Hinge loss	max(0, 1 − y·ŷ)	Used by support-vector-style classifiers

(Cross-entropy is explained properly in Section 6.)

Choosing a loss: the loss should reflect what you actually care about. Squared error punishes outliers harshly, absolute error shrugs them off. It also needs to be differentiable (or nearly so) for gradient-based methods.

Regularization (adding a penalty)

Add a term to the objective to discourage overly complex models:

Total objective = Loss + λ · Penalty(θ)

• L2 (Ridge): penalty = Σ θ² (shrinks weights smoothly)
• L1 (Lasso): penalty = Σ |θ| (pushes some weights to exactly zero)

5.3 Convexity

Convex sets and functions

• A convex set: for any two points in the set, the straight line between them also lies in the set.
• A convex function: the line segment between any two points on its graph lies on or above the graph.

Mathematically, f is convex if:
f(λx + (1−λ)y)  ≤  λ·f(x) + (1−λ)·f(y)     for all x, y and 0 ≤ λ ≤ 1

  Convex (bowl)            Non-convex (bumpy)
    \         /              \    /\    /
     \       /                \  /  \  /
      \_____/                  \/    \/
   ONE valley               MANY valleys

Why convexity is a big deal

For a convex function, every local minimum is a global minimum.

So gradient descent (with a suitable step size) is guaranteed to reach the best answer. No getting trapped in a bad valley.

How to check

• One variable: f″(x) ≥ 0 everywhere.
• Many variables: the Hessian is positive semi-definite everywhere.
• Strictly convex: gives a unique minimum.

Examples

• ✅ Convex: MSE for linear regression, logistic regression's log loss, x², eˣ, |x|
• ❌ Non-convex: loss surfaces of deep neural networks (full of local minima and saddle points)

Even though deep learning problems are non-convex, gradient-based methods still work well in practice.

5.4 Gradient Descent (GD)

The workhorse of ML. Walk downhill: repeatedly step in the direction opposite to the gradient.

Update rule

θ_new = θ_old − η · ∇J(θ_old)

• θ = parameters
• η (eta), also called α = learning rate (step size)
• ∇J = gradient of the cost function

Algorithm

1. Initialise parameters (often randomly).
2. Compute the gradient of the cost using the entire dataset.
3. Update: θ ← θ − η∇J(θ).
4. Repeat until the cost stops improving (convergence) or a set number of iterations is reached.

Mini example

Minimise f(w) = w² (gradient 2w), start at w = 4, learning rate η = 0.1:

Step	w	gradient 2w	new w = w − 0.1·(2w)
0	4.00	8.00	3.20
1	3.20	6.40	2.56
2	2.56	5.12	2.05
…	…	…	→ 0 (the minimum)

The learning rate matters enormously

Learning rate	What happens
Too small	Converges, but painfully slowly
Just right	Smooth, efficient descent
Too large	Overshoots, bounces around or even diverges (error explodes)

Practical notes

• Feature scaling (making features similar in range) makes gradient descent converge much faster.
• Learning rate schedules (decreasing η over time) help settle into a minimum.
• In non-convex problems it can get stuck at local minima or saddle points or crawl across plateaus.

5.5 Stochastic Gradient Descent (SGD)

Plain (batch) gradient descent uses all the data for each step, which is slow for big datasets. SGD uses just one randomly chosen sample per update.

For each sample (xᵢ, yᵢ):
    θ ← θ − η · ∇Lᵢ(θ)

Pros

• ⚡ Very fast per step and cheap on memory
• Handles huge datasets and online/streaming data
• The noise in the updates can help escape shallow local minima and saddle points

Cons

• The path is noisy and zig-zaggy because one sample is a rough estimate of the true gradient
• It tends to bounce around the minimum rather than settling exactly (fixed by decaying the learning rate)
• Can't take full advantage of vectorised hardware

Epoch: one full pass through the entire training set. Data is usually shuffled each epoch.

5.6 Mini-Batch Gradient Descent

The sweet spot between the two extremes: compute the gradient on small random groups of samples (the mini-batch, e.g., 32, 64, 128 or 256).

θ ← θ − η · (1/m) Σ(over m samples in batch) ∇Lᵢ(θ)

Variant	Samples per update	Speed per update	Path smoothness	Memory
Batch GD	All N	Slow	Smooth and stable	High
Stochastic GD	1	Very fast	Very noisy	Low
Mini-Batch GD	m (e.g., 32–256)	Fast	Moderately smooth	Moderate

Why it's the default in practice

• Gradient estimate is less noisy than SGD.
• Exploits matrix operations / GPUs efficiently.
• Good balance of speed and stability.

Terminology

• Batch size = samples per update
• Iteration = one parameter update
• Epoch = one full pass over the data (N / batch size iterations)

ℹ️ In many libraries and papers, "SGD" loosely refers to mini-batch SGD.

Common improvements to plain gradient descent (names only)

Momentum (builds up speed in consistent directions), RMSprop, Adam (adapts the step size per parameter). These are all refinements of the same core idea.

5.7 Constrained Optimization

Sometimes the solution must obey rules. We minimise an objective subject to constraints.

minimize    f(x)
subject to  gᵢ(x) ≤ 0      (inequality constraints)
            hⱼ(x) = 0      (equality constraints)

The set of points satisfying all constraints is the feasible region.

Example: Minimise x² + y² subject to x + y = 1. (Find the point closest to the origin that lies on the line x + y = 1. The answer is (0.5, 0.5).)

Lagrange Multipliers (equality constraints)

Combine the objective and constraints into one function, the Lagrangian:

L(x, λ) = f(x) + λ · h(x)

Set all the partial derivatives to zero: ∇ₓL = 0 and ∂L/∂λ = 0 (which just recovers the constraint).

Worked example: minimise f = x² + y² subject to x + y − 1 = 0.

L = x² + y² + λ(x + y − 1)
∂L/∂x = 2x + λ = 0   →  x = −λ/2
∂L/∂y = 2y + λ = 0   →  y = −λ/2
x + y = 1            →  −λ = 1  →  λ = −1
⇒  x = y = 0.5

Intuition: At the optimum, the gradient of the objective is parallel to the gradient of the constraint. You can't improve without breaking the constraint. The multiplier λ measures how sensitive the optimal value is to the constraint.

Inequality constraints and the KKT conditions

For inequality constraints, the Karush–Kuhn–Tucker (KKT) conditions generalise Lagrange multipliers. At a constrained optimum:

1. Stationarity: the gradient of the Lagrangian is zero.
2. Primal feasibility: all constraints are satisfied.
3. Dual feasibility: multipliers for inequality constraints are ≥ 0.
4. Complementary slackness: each multiplier × its constraint = 0. An inequality constraint is either active (touching the boundary) or its multiplier is zero (not affecting the solution).

For convex problems, the KKT conditions are both necessary and sufficient for optimality.

Other practical ways to handle constraints

• Projected gradient descent: take a normal gradient step, then project the result back onto the feasible region.
• Penalty methods: add a large penalty for violating a constraint into the objective.

🧠 Why ML cares

Constraints appear when probabilities must sum to 1, when weights must be non-negative, or when limiting model size. Regularization can also be viewed as a constrained problem (e.g., "keep the weights' norm below some budget").

---

6. Information Theory

The big idea: Information theory gives us a way to measure uncertainty and surprise. It's behind the loss functions used in classification and the splitting rules of decision trees.

Intuition: A rare, surprising event carries more information than a common, expected one. "The sun rose today" tells you nothing. "It snowed in the desert" tells you a lot.

6.1 Information Content (Surprise)

The information (or surprise) of an event with probability p:

I(x) = −log₂ p(x)        (measured in bits)

• Certain event (p = 1): I = 0 bits (no surprise)
• Fair coin lands heads (p = 0.5): I = 1 bit
• Event with p = 1/8: I = 3 bits

(Using natural log gives units of nats instead of bits.)

6.2 Entropy

Entropy is the average surprise (average information) of a random variable. It measures the uncertainty or impurity/disorder in a distribution.

H(X) = − Σ p(x) · log₂ p(x)

(By convention, 0 · log 0 = 0.)

Examples

Distribution	Entropy	Why
Fair coin (0.5, 0.5)	1 bit	Maximum uncertainty for 2 outcomes
Biased coin (0.9, 0.1)	−0.9·log₂0.9 − 0.1·log₂0.1 ≈ 0.469 bits	More predictable
Certain outcome (1, 0)	0 bits	No uncertainty
Fair 8-sided die	3 bits	log₂ 8

Key facts

• H(X) ≥ 0
• Minimum (0) when one outcome is certain.
• Maximum when all outcomes are equally likely (uniform): log₂ n for n outcomes.
• Interpretation: the minimum average number of bits needed to encode outcomes drawn from the distribution.

Entropy
  1 │        ___
    │      .'   '.
    │    .'       '.
    │  .'           '.
  0 └──┴─────┴───────┴── p (probability of heads)
     0      0.5       1
         (peak at fair coin)

6.3 Cross-Entropy

Suppose the true distribution is P but you use a model's predicted distribution Q. Cross-entropy is the average number of bits needed to encode data from P when using a code optimised for Q.

H(P, Q) = − Σ p(x) · log q(x)

Key points

• H(P, Q) ≥ H(P), and equality holds only when Q = P.
• It's not symmetric: H(P, Q) ≠ H(Q, P).
• The extra cost = KL divergence: H(P, Q) = H(P) + D_KL(P ‖ Q).

🧠 Why ML cares: the classification loss

For classification, P is the true label distribution (one-hot: 1 for the correct class, 0 elsewhere) and Q is the model's predicted probabilities. The cross-entropy loss then simplifies to:

Loss = − log( predicted probability of the correct class )

Example: True class is "cat". Model predicts [cat 0.7, dog 0.2, bird 0.1]. Loss = −ln(0.7) ≈ 0.357. If it had predicted cat with 0.1, the loss would be −ln(0.1) ≈ 2.303, a much bigger penalty for being confidently wrong.

Binary version: −[y·log(ŷ) + (1−y)·log(1−ŷ)].

Minimising cross-entropy is equivalent to maximising the likelihood of the data under the model.

6.4 KL Divergence (Kullback–Leibler Divergence)

Measures how different one probability distribution Q is from a reference distribution P. It's the "extra bits" wasted by using Q instead of P.

D_KL(P ‖ Q) = Σ p(x) · log( p(x) / q(x) )
            = H(P, Q) − H(P)

Properties

• D_KL(P ‖ Q) ≥ 0 always.
• D_KL(P ‖ Q) = 0 only if P = Q.
• ⚠️ Not symmetric: D_KL(P ‖ Q) ≠ D_KL(Q ‖ P), so it's a "divergence", not a true distance.
• Infinite if Q gives zero probability to something P considers possible.

Why it's useful

• Since H(P) is fixed by the data, minimising cross-entropy = minimising KL divergence between the true and predicted distributions.
• Used to measure how close a model's distribution is to the truth, and in variational methods (as a regulariser that keeps one distribution close to another).

6.5 Mutual Information

Measures how much knowing one variable reduces uncertainty about another: how much information X and Y share.

I(X; Y) = H(X) − H(X | Y)
        = H(X) + H(Y) − H(X, Y)
        = D_KL( P(X,Y) ‖ P(X)·P(Y) )

Properties

• I(X; Y) ≥ 0
• I(X; Y) = 0 if and only if X and Y are independent
• Symmetric: I(X; Y) = I(Y; X)
• Higher value means stronger dependence

Intuition: If knowing today's cloud cover greatly reduces your uncertainty about rain, then cloud cover and rain have high mutual information.

Compared with correlation

	Correlation	Mutual information
Detects	Linear relationships only	Any kind of dependency (linear or non-linear)
Range	−1 to +1	0 to ∞
Direction (+/−)	Yes	No

🧠 Why ML cares

Used for feature selection (keep features that share a lot of information with the target) and appears in the theory behind decision tree splitting.

6.6 Information Gain

Information gain is the reduction in entropy achieved by splitting data on a feature. It's the key criterion for choosing splits in decision trees.

IG(S, A) = H(S) − Σ ( |Sᵥ| / |S| ) · H(Sᵥ)

where S is the dataset, A is the feature we split on, and each Sᵥ is the subset for one value of A. The sum is the weighted average entropy after the split.

(Information gain is exactly the mutual information between the feature and the label.)

Worked example

Parent node: 10 samples, 5 positive / 5 negative, so H = 1 bit.

Split on a feature into:

• Left (4 samples): 4 positive, 0 negative, so H = 0
• Right (6 samples): 1 positive, 5 negative, so H = −(1/6)log₂(1/6) − (5/6)log₂(5/6) ≈ 0.650

Weighted entropy after split = (4/10)(0) + (6/10)(0.650) ≈ 0.390

IG = 1 − 0.390 = 0.610 bits

A high gain means the split made the groups much purer. The tree picks the feature with the highest information gain.

6.7 Gini Impurity

An alternative measure of node impurity used by decision trees (e.g., the CART algorithm). It's the probability of misclassifying a randomly chosen element if you labelled it randomly according to the node's class distribution.

Gini = 1 − Σ pₖ²

Node composition	Gini
100% one class (pure)	1 − 1² = 0
50% / 50% (binary, most impure)	1 − (0.25 + 0.25) = 0.5
90% / 10%	1 − (0.81 + 0.01) = 0.18

• Range (binary case): 0 (pure) to 0.5 (worst). In general, the max is 1 − 1/k for k classes.
• A split is chosen to minimise the weighted Gini of the child nodes (equivalently, maximise the Gini decrease).

Gini vs Entropy

	Entropy	Gini Impurity
Formula	−Σ p log₂ p	1 − Σ p²
Binary max value	1	0.5
Computation	Uses a logarithm (slightly slower)	Simple squares (slightly faster)
Results in practice	Very similar trees in most cases	Very similar trees in most cases
Used with	Information gain (ID3, C4.5)	CART

Information-theory family at a glance

Entropy H(P)          → uncertainty in one distribution
Cross-Entropy H(P,Q)  → cost of using Q to describe P  = H(P) + KL
KL Divergence         → extra cost / difference between P and Q
Mutual Information    → shared information between X and Y
Information Gain      → entropy reduction from a split (= mutual info of feature & label)
Gini Impurity         → alternative to entropy for measuring node impurity

---

7. Quick Revision Cheat Sheet

🔷 Linear Algebra

Concept	One-liner
Scalar / Vector / Matrix / Tensor	0-D / 1-D / 2-D / n-D arrays of numbers
Matrix multiply	(m×n)·(n×p) = (m×p); not commutative
Transpose	Swap rows and columns; (AB)ᵀ = BᵀAᵀ
Inverse	A·A⁻¹ = I; exists only if square and det ≠ 0
Determinant	Volume-scaling factor; 0 means singular
Rank	Number of independent rows/columns
Dot product	a·b = ‖a‖‖b‖cosθ; 0 means orthogonal
Eigen	Av = λv; directions that only get stretched
SVD	A = UΣVᵀ; works for any matrix
Vector space	Closed under addition and scaling; has a basis and dimension

🔷 Calculus

Concept	One-liner
Limit	Value a function approaches
Derivative	Slope / instantaneous rate of change
Partial derivative	Derivative w.r.t. one variable, others fixed
Chain rule	dy/dx = (dy/du)(du/dx); the basis of backpropagation
Gradient	Vector of partials; points to steepest increase
Jacobian	Matrix of first derivatives (vector → vector function)
Hessian	Matrix of second derivatives (curvature)

🔷 Probability

Concept	One-liner
Conditional	P(A|B) = P(A∩B)/P(B)
Bayes	P(A|B) = P(B|A)P(A)/P(B)
Independence	P(A∩B) = P(A)P(B)
Expectation	Probability-weighted average
Variance	E[X²] − (E[X])²
Covariance	How two variables move together
Gaussian	Bell curve; 68–95–99.7 rule

🔷 Statistics

Concept	One-liner
Mean / Median / Mode	Average / middle / most frequent
Std deviation	Typical distance from the mean; sample uses n−1
Correlation	Standardised covariance in [−1, 1]; ≠ causation
Population vs sample	Whole group vs subset; parameters vs statistics
Hypothesis test	Reject H₀ if p ≤ α
Confidence interval	Range of plausible values for a parameter
p-value	P(data this extreme | H₀ true), not P(H₀ true)

🔷 Optimization

Concept	One-liner
Objective / loss	What we minimise / error on predictions
Convex	Any local minimum is the global minimum
Gradient descent	θ ← θ − η∇J; uses all data
SGD	One sample per update; fast but noisy
Mini-batch GD	Small batches; the practical default
Constrained optimization	Lagrange multipliers and KKT conditions

🔷 Information Theory

Concept	Formula
Entropy	H = −Σ p log p
Cross-entropy	H(P,Q) = −Σ p log q
KL divergence	Σ p log(p/q) (not symmetric, ≥ 0)
Mutual information	H(X) − H(X|Y)
Information gain	Parent entropy − weighted child entropy
Gini impurity	1 − Σ p²

---

✅ Final Takeaways

1. Linear algebra is how data and models are stored and transformed.
2. Calculus tells us how to nudge parameters to reduce error (gradients and the chain rule).
3. Probability models uncertainty. Bayes' theorem updates beliefs with data.
4. Statistics lets us draw careful conclusions from limited samples.
5. Optimization turns all of this into an algorithm: follow the gradient downhill.
6. Information theory supplies the loss functions (cross-entropy) and split criteria (entropy, Gini) used all over ML.

---

Friendly, beginner-to-advanced notes on the Python skills and libraries you need for Machine Learning. Every section has short, runnable code, plain-English explanations, and a note on why ML cares.

---

📑 Table of Contents

1. Python Fundamentals and Functions
2. Object-Oriented Programming Basics
3. NumPy: Arrays and Vectorization
4. Pandas: Series and DataFrames
5. Data Cleaning and Manipulation
6. Matplotlib and Seaborn Visualization
7. SciPy for Scientific Computing
8. Scikit-learn Fundamentals
9. Jupyter Notebook and Google Colab
10. Virtual Environments and Package Management
11. Quick Revision Cheat Sheet

🗺️ How the pieces fit together

 Python (language)        → the glue that holds everything together
 NumPy                    → fast numeric arrays (the foundation)
 Pandas                   → tables of data built on NumPy
 Matplotlib / Seaborn     → see your data
 SciPy                    → scientific algorithms (stats, optimization, ...)
 Scikit-learn             → machine learning models and tools
 Jupyter / Colab          → where you write, run and explain code interactively
 Virtual environments     → keep each project's packages tidy and separate

A typical ML workflow touches all of them:

Load data (Pandas) → Clean (Pandas) → Explore (Seaborn/Matplotlib) → Prepare arrays (NumPy)
        → Train & evaluate (Scikit-learn) → Share results (Jupyter)

---

1. Python Fundamentals and Functions

Python is the most popular language for ML because it is easy to read, has a huge ecosystem of libraries, and lets you go from idea to experiment quickly.

1.1 Variables and Data Types

A variable is a name that points to a value. Python figures out the type automatically (dynamic typing).

age = 25                 # int
height = 1.75            # float
name = "Asha"            # str
is_student = True        # bool
nothing = None           # NoneType (means "no value")

print(type(age))         # <class 'int'>

Type	Example	Notes
int	42	Whole numbers (no size limit)
float	3.14	Decimal numbers
str	"hello"	Text
bool	True, False	Logical values
NoneType	None	Represents "nothing"

Type conversion (casting): int("5"), float(3), str(10), bool(0) (→ False).

1.2 Operators

Kind	Operators	Example
Arithmetic	+ - * / // % **	7 // 2 = 3, 7 % 2 = 1, 2 ** 3 = 8, 7 / 2 = 3.5
Comparison	== != < > <= >=	5 >= 3 → True
Logical	and, or, not	(x > 0) and (x < 10)
Membership	in, not in	"a" in "cat" → True
Identity	is, is not	x is None
Assignment	= += -= *= /=	x += 1

1.3 Strings

s = "Machine Learning"
s.lower()              # 'machine learning'
s.upper()              # 'MACHINE LEARNING'
s.split(" ")           # ['Machine', 'Learning']
s.replace("Machine", "Deep")
s[0]                   # 'M'   (indexing starts at 0)
s[0:7]                 # 'Machine'   (slicing)
len(s)                 # 16

# f-strings: the modern way to format text
accuracy = 0.93456
print(f"Accuracy: {accuracy:.2%}")      # Accuracy: 93.46%
print(f"Model: {s}, score={accuracy:.3f}")

1.4 Core Data Structures

Structure	Syntax	Ordered?	Mutable?	Duplicates?	Use it for
List	[1, 2, 3]	✅	✅	✅	General-purpose sequences
Tuple	(1, 2, 3)	✅	❌	✅	Fixed data, e.g. a shape (100, 5)
Dictionary	{"a": 1}	✅ (insertion order)	✅	Keys unique	Key → value lookups, hyperparameters
Set	{1, 2, 3}	❌	✅	❌	Unique items, fast membership checks

# List
nums = [10, 20, 30]
nums.append(40)            # [10, 20, 30, 40]
nums[-1]                   # 40  (negative index = from the end)
nums[1:3]                  # [20, 30]

# Tuple
shape = (100, 5)
rows, cols = shape         # unpacking

# Dictionary
params = {"lr": 0.01, "epochs": 50}
params["batch_size"] = 32
params.get("momentum", 0.9)      # safe lookup with default
for key, value in params.items():
    print(key, value)

# Set
labels = {"cat", "dog", "cat"}   # {'cat', 'dog'}

1.5 Control Flow

# if / elif / else
score = 82
if score >= 90:
    grade = "A"
elif score >= 75:
    grade = "B"
else:
    grade = "C"

# for loop
for i in range(3):               # 0, 1, 2
    print(i)

for idx, label in enumerate(["cat", "dog"]):
    print(idx, label)            # 0 cat / 1 dog

for a, b in zip([1, 2], ["x", "y"]):
    print(a, b)

# while loop
n = 0
while n < 3:
    n += 1

# break / continue
for x in range(10):
    if x == 2: continue          # skip 2
    if x == 5: break             # stop at 5

⚠️ Python uses indentation (spaces) to define blocks. Be consistent. 4 spaces is the standard.

1.6 Comprehensions: Compact Loops

squares = [x**2 for x in range(5)]               # [0, 1, 4, 9, 16]
evens   = [x for x in range(10) if x % 2 == 0]   # [0, 2, 4, 6, 8]
lookup  = {w: len(w) for w in ["cat", "horse"]}  # {'cat': 3, 'horse': 5}
unique  = {x % 3 for x in range(10)}             # {0, 1, 2}

1.7 Functions

A function is a reusable block of code. It keeps programs organised and avoids repetition (DRY: Don't Repeat Yourself).

def mean_squared_error(y_true, y_pred):
    """Return the average squared difference between two lists."""
    total = 0
    for t, p in zip(y_true, y_pred):
        total += (t - p) ** 2
    return total / len(y_true)

mean_squared_error([3, 5], [2, 7])       # 2.5

The text in triple quotes is a docstring, the function's built-in documentation (help(mean_squared_error) shows it).

Parameters and arguments

def train(model_name, epochs=10, lr=0.01):    # epochs and lr have defaults
    return f"{model_name}: epochs={epochs}, lr={lr}"

train("cnn")                         # uses defaults
train("cnn", 20)                     # positional arguments
train("cnn", lr=0.1, epochs=5)       # keyword arguments (order doesn't matter)

*args and **kwargs: flexible numbers of arguments

def total(*args):                    # args is a tuple of all positional args
    return sum(args)
total(1, 2, 3)                       # 6

def describe(**kwargs):              # kwargs is a dict of all keyword args
    for k, v in kwargs.items():
        print(k, "=", v)
describe(lr=0.01, epochs=5)

Many ML libraries accept **kwargs to pass settings along.

Returning multiple values

def min_max(values):
    return min(values), max(values)   # actually returns a tuple

low, high = min_max([4, 9, 1])

Lambda (anonymous) functions

square = lambda x: x ** 2
square(4)                            # 16

pairs = [("a", 3), ("b", 1), ("c", 2)]
sorted(pairs, key=lambda p: p[1])    # sort by the second item

map and filter

list(map(lambda x: x * 2, [1, 2, 3]))        # [2, 4, 6]
list(filter(lambda x: x > 1, [1, 2, 3]))     # [2, 3]

(List comprehensions are usually more readable.)

Scope: where a variable lives

x = 10                 # global

def f():
    x = 5              # local; separate from the global x
    return x

f()                    # 5
x                      # still 10

Variables created inside a function are local to it.

⚠️ Pitfall: mutable default arguments

def bad(item, bucket=[]):      # the SAME list is reused on every call!
    bucket.append(item)
    return bucket

def good(item, bucket=None):
    if bucket is None:
        bucket = []
    bucket.append(item)
    return bucket

1.8 Modules and Imports

A module is a .py file of reusable code. A package is a folder of modules. Libraries are imported with import.

import math
import numpy as np                   # alias: np
from math import sqrt                # import just one thing
from sklearn.model_selection import train_test_split

math.sqrt(16)

1.9 Error Handling

try:
    value = int("abc")
except ValueError as e:
    print("Could not convert:", e)
else:
    print("Success")                 # runs only if no error occurred
finally:
    print("Always runs")             # cleanup code

Common errors: ValueError, TypeError, KeyError, IndexError, FileNotFoundError.

1.10 Reading and Writing Files

with open("notes.txt", "w") as f:        # 'with' closes the file automatically
    f.write("hello\n")

with open("notes.txt", "r") as f:
    text = f.read()

1.11 Generators (a handy advanced idea)

A generator produces values one at a time instead of building a whole list in memory, which is useful for large data.

def batches(data, size):
    for i in range(0, len(data), size):
        yield data[i:i + size]           # 'yield' pauses and resumes

for batch in batches(list(range(10)), 4):
    print(batch)       # [0,1,2,3] / [4,5,6,7] / [8,9]

🧠 Why ML cares

Data loading, feeding data in batches, writing helper functions for preprocessing, and organising experiments all rely on these basics.

---

2. Object-Oriented Programming Basics

Object-Oriented Programming (OOP) organises code into objects that bundle data (attributes) and behaviour (methods). ML libraries are built this way. A model is an object with methods like .fit() and .predict().

2.1 Classes and Objects

• Class: a blueprint (e.g., "Dog").
• Object (instance): a specific thing built from that blueprint (e.g., my dog Bruno).

class Student:
    school = "ABC University"            # class attribute (shared by all instances)

    def __init__(self, name, marks):     # constructor: runs when the object is created
        self.name = name                 # instance attributes (unique to each object)
        self.marks = marks

    def is_passing(self):                # method
        return self.marks >= 40

    def __str__(self):                   # what print() shows
        return f"{self.name} ({self.marks})"

s1 = Student("Asha", 72)
s2 = Student("Ravi", 35)
s1.is_passing()                          # True
print(s2)                                # Ravi (35)

Term	Meaning
__init__	Special method (constructor) that sets up a new object
self	Refers to the current object. It's always the first parameter of a method
Attribute	A variable stored on an object (s1.name)
Method	A function defined inside a class (s1.is_passing())
Instance attribute	Belongs to one object
Class attribute	Shared by all objects of the class

2.2 The Four Pillars of OOP

1) Encapsulation: bundle data with methods, and control access

Python uses naming conventions:

• name: public
• _name: "internal use" (a polite hint, not enforced)
• __name: name-mangled to make accidental access harder

class Account:
    def __init__(self, balance):
        self._balance = balance           # internal

    @property
    def balance(self):                    # read-only access via a property
        return self._balance

    def deposit(self, amount):
        if amount <= 0:
            raise ValueError("Amount must be positive")
        self._balance += amount

a = Account(100)
a.deposit(50)
a.balance                                  # 150

2) Inheritance: a new class reuses and extends an existing one

class Animal:
    def __init__(self, name):
        self.name = name
    def speak(self):
        return "..."

class Dog(Animal):                         # Dog inherits from Animal
    def speak(self):                       # overrides the parent method
        return "Woof!"

class Cat(Animal):
    def __init__(self, name, indoor):
        super().__init__(name)             # call the parent's constructor
        self.indoor = indoor
    def speak(self):
        return "Meow!"

3) Polymorphism: the same method name, different behaviour

for pet in [Dog("Bruno"), Cat("Mia", True)]:
    print(pet.name, pet.speak())           # each class responds its own way

This is why you can swap one ML model for another, since they all share .fit() and .predict().

4) Abstraction: expose what's needed, hide the details

You call model.fit(X, y) without knowing the math inside.

2.3 Special ("Dunder") Methods

Methods with double underscores customise how objects behave.

Method	Purpose	Triggered by
__init__	Initialise object	Student(...)
__str__	Friendly text	print(obj)
__repr__	Developer-oriented text	Typing the object in a console
__len__	Length	len(obj)
__getitem__	Indexing	obj[0]
__add__	The + operator	a + b
__call__	Make an object callable	obj(...)

2.4 Class Methods and Static Methods

class Dataset:
    def __init__(self, rows):
        self.rows = rows

    @classmethod
    def from_csv_text(cls, text):          # alternative constructor; gets the class (cls)
        return cls(text.split("\n"))

    @staticmethod
    def is_valid_row(row):                 # no self / cls; just a utility function
        return len(row) > 0

2.5 OOP in ML: A Mini "Estimator"

Here's a toy model that follows the same pattern as Scikit-learn's estimators:

class MeanRegressor:
    """Predicts the average of the training targets. Simple, but shows the pattern."""

    def fit(self, X, y):
        self.mean_ = sum(y) / len(y)       # learned attribute (trailing underscore = learned)
        return self                        # returning self allows chaining

    def predict(self, X):
        return [self.mean_ for _ in X]

model = MeanRegressor()
model.fit([[1], [2], [3]], [10, 20, 30])
model.predict([[4], [5]])                  # [20.0, 20.0]

🧠 Why ML cares

• Scikit-learn models, PyTorch/TensorFlow models and Pandas DataFrames are all objects.
• Writing your own reusable preprocessing steps or models means writing classes.
• Understanding self, inheritance and methods makes library documentation far easier to read.

---

3. NumPy: Arrays and Vectorization

NumPy (Numerical Python) provides the ndarray, a fast, memory-efficient N-dimensional array. It's the foundation of almost the entire Python ML ecosystem (Pandas, SciPy and Scikit-learn all build on it).

import numpy as np

3.1 Why NumPy Instead of Python Lists?

	Python list	NumPy array
Element types	Mixed allowed	One type (homogeneous)
Speed	Slow loops	Fast (compiled C under the hood)
Memory	Heavier	Compact
Math on whole collection	Needs loops	Built-in vectorized operations
Dimensions	Nested lists	True N-D arrays

3.2 Creating Arrays

a = np.array([1, 2, 3])                    # 1-D
M = np.array([[1, 2, 3], [4, 5, 6]])       # 2-D (2 rows, 3 columns)

np.zeros((2, 3))                           # 2×3 of 0s
np.ones((2, 3))                            # 2×3 of 1s
np.full((2, 2), 7)                         # filled with 7
np.eye(3)                                  # 3×3 identity matrix
np.arange(0, 10, 2)                        # [0 2 4 6 8]
np.linspace(0, 1, 5)                       # 5 evenly spaced values from 0 to 1

rng = np.random.default_rng(42)            # seeded generator = reproducible results
rng.random((2, 3))                         # uniform random in [0, 1)
rng.normal(0, 1, size=5)                   # Gaussian samples
rng.integers(0, 10, size=4)                # random ints

🔁 Always set a seed in ML experiments so results can be reproduced.

3.3 Array Attributes

M.shape       # (2, 3)  → rows, columns
M.ndim        # 2       → number of dimensions
M.size        # 6       → total elements
M.dtype       # int64   → data type
M.T           # transpose

Data types: int32, int64, float32, float64, bool. Convert with M.astype(np.float32). (float32 halves memory vs float64, which is common in deep learning.)

3.4 Indexing and Slicing

M = np.array([[10, 20, 30],
              [40, 50, 60],
              [70, 80, 90]])

M[0, 1]          # 20            → row 0, column 1
M[1]             # [40 50 60]    → entire row 1
M[:, 2]          # [30 60 90]    → entire column 2
M[0:2, 1:3]      # [[20 30], [50 60]]   → sub-matrix
M[-1, -1]        # 90

Format: array[row_slice, column_slice], with slices as start:stop:step.

Boolean masking: select by condition

x = np.array([5, -2, 8, -1, 3])
x > 0                    # [ True False  True False  True]
x[x > 0]                 # [5 8 3]
x[x < 0] = 0             # replace negatives with 0 → [5 0 8 0 3]

Fancy indexing: select by a list of positions

x = np.array([10, 20, 30, 40])
x[[0, 2]]                # [10 30]

⚠️ Slices are views, not copies. Changing a slice changes the original. Use .copy() for an independent copy.

b = a[:2].copy()

3.5 Reshaping and Combining

a = np.arange(12)              # [0 1 2 ... 11]
a.reshape(3, 4)                # 3 rows × 4 columns
a.reshape(2, -1)               # -1 means "figure this dimension out" → 2×6
a.flatten()                    # back to 1-D

x = np.array([1, 2, 3])
x[:, np.newaxis].shape         # (3, 1)  → add a dimension (column vector)

np.concatenate([a1, a2])       # join along an existing axis
np.vstack([a1, a2])            # stack vertically (as rows)
np.hstack([a1, a2])            # stack horizontally (as columns)
np.stack([a1, a2])             # join along a NEW axis

3.6 Vectorization: The Big Idea ⚡

Vectorization means applying an operation to an entire array at once, with no explicit Python loop. NumPy runs the loop in fast compiled code.

x = np.array([1, 2, 3, 4])

# ❌ Slow, loop style
result = []
for v in x:
    result.append(v * 2 + 1)

# ✅ Fast, vectorized style
result = x * 2 + 1               # [3 5 7 9]

Universal functions (ufuncs) apply element-wise:

np.sqrt(x)      np.exp(x)      np.log(x)      np.sin(x)      np.abs(x)

Vectorized math feels like the formulas:

# Mean Squared Error: (1/n) Σ (y − ŷ)²
mse = np.mean((y_true - y_pred) ** 2)

# Sigmoid activation
sigmoid = 1 / (1 + np.exp(-z))

Speed intuition: for a million numbers, the vectorized version is typically tens to hundreds of times faster than a Python loop.

3.7 Broadcasting: Math Between Different Shapes

Broadcasting lets NumPy combine arrays of different shapes by virtually "stretching" the smaller one, without copying data.

M = np.array([[1, 2, 3],
              [4, 5, 6]])         # shape (2, 3)
row = np.array([10, 20, 30])      # shape (3,)

M + row
# [[11 22 33]
#  [14 25 36]]    ← the row was added to every row of M

The rules (compare shapes from the rightmost dimension backwards). Two dimensions are compatible if they are equal or one of them is 1. Missing dimensions are treated as 1.

(2, 3) + (3,)    → ✅ (3,) acts like (1, 3) → stretched to (2, 3)
(3, 1) + (1, 4)  → ✅ result (3, 4)
(2, 3) + (2,)    → ❌ 3 vs 2 don't match

Practical example: standardising features

X = np.array([[1., 200.],
              [2., 300.],
              [3., 400.]])

X_scaled = (X - X.mean(axis=0)) / X.std(axis=0)
# means and stds have shape (2,), broadcast across every row of X

3.8 Aggregations and the axis Argument

M = np.array([[1, 2, 3],
              [4, 5, 6]])

M.sum()            # 21        → all elements
M.sum(axis=0)      # [5 7 9]   → collapse ROWS → one value per COLUMN
M.sum(axis=1)      # [6 15]    → collapse COLUMNS → one value per ROW
M.mean(), M.std(), M.min(), M.max(), M.argmax(), np.median(M)

🧠 Remember: axis=0 goes down (across rows), axis=1 goes across (across columns). The axis you name is the one that disappears.

3.9 Linear Algebra with NumPy

A = np.array([[1, 2], [3, 4]])
B = np.array([[5, 6], [7, 8]])
v = np.array([1, 2])

A * B                    # element-wise multiplication
A @ B                    # matrix multiplication → [[19 22], [43 50]]
np.dot(v, v)             # dot product → 5
A.T                      # transpose

np.linalg.inv(A)         # inverse
np.linalg.det(A)         # determinant → -2.0
np.linalg.matrix_rank(A) # rank
vals, vecs = np.linalg.eig(A)       # eigenvalues and eigenvectors
U, S, Vt = np.linalg.svd(A)         # singular value decomposition
np.linalg.norm(v)        # vector length
np.linalg.solve(A, b)    # solve A x = b  (preferred over inv(A) @ b)

3.10 Handy Extras

np.where(x > 0, "pos", "neg")      # vectorized if-else
np.unique(labels, return_counts=True)
np.sort(x), np.argsort(x)          # sorted values / sorted positions
np.clip(x, 0, 1)                   # limit values to a range
np.isnan(x)                        # find missing values (NaN)
np.cumsum(x)                       # running total

🧠 Why ML cares

Datasets become NumPy arrays X with shape (n_samples, n_features). Predictions, weights, gradients and loss calculations are all vectorized array operations. Writing loops over samples is almost always the wrong choice.

---

4. Pandas: Series and DataFrames

Pandas is the go-to library for tabular data (think Excel spreadsheet or SQL table) with labelled rows and columns.

import pandas as pd

4.1 Series: A Labelled 1-D Array

A Series is a single column of data with an index (labels).

s = pd.Series([250, 300, 150], index=["Mon", "Tue", "Wed"], name="sales")

s["Tue"]            # 300      → by label
s.iloc[0]           # 250      → by position
s[s > 200]          # Mon, Tue → boolean filter
s * 2               # vectorized, like NumPy
s.mean(), s.max(), s.describe()

4.2 DataFrame: A 2-D Table

A DataFrame is a table of columns, where each column is a Series (and each can have a different type).

df = pd.DataFrame({
    "name":   ["Asha", "Ravi", "Meera", "Karan"],
    "age":    [25, 31, 28, 40],
    "city":   ["Bhopal", "Delhi", "Bhopal", "Pune"],
    "salary": [50000, 72000, 61000, 90000],
})

    name  age    city  salary
0   Asha   25  Bhopal   50000
1   Ravi   31   Delhi   72000
2  Meera   28  Bhopal   61000
3  Karan   40    Pune   90000

Parts of a DataFrame: df.index (row labels), df.columns (column names), df.values (the underlying NumPy array), df.shape, df.dtypes.

4.3 Reading and Writing Data

df = pd.read_csv("data.csv")             # most common
df = pd.read_excel("data.xlsx")
df = pd.read_json("data.json")
df = pd.read_csv("data.csv", sep=";", usecols=["age", "city"], nrows=1000)

df.to_csv("clean.csv", index=False)      # index=False avoids an extra index column

4.4 First Look at Your Data

df.head()           # first 5 rows
df.tail(3)          # last 3 rows
df.shape            # (rows, columns)
df.info()           # column names, types, non-null counts, memory use
df.describe()       # count, mean, std, min, quartiles, max for numeric columns
df.describe(include="object")   # summary of text columns
df.columns          # column names
df["city"].value_counts()       # frequency of each category
df["city"].unique()             # distinct values
df.nunique()                    # number of unique values per column

4.5 Selecting Data

df["age"]                    # one column → Series
df[["name", "age"]]          # several columns → DataFrame

# By LABEL with .loc[rows, columns]
df.loc[0, "name"]                          # 'Asha'
df.loc[0:2, ["name", "age"]]               # label slices INCLUDE the end (rows 0,1,2)

# By POSITION with .iloc[rows, columns]
df.iloc[0, 1]                              # row 0, column 1 → 25
df.iloc[0:2, 0:2]                          # position slices EXCLUDE the end

# Boolean filtering
df[df["age"] > 28]
df[(df["age"] > 28) & (df["city"] == "Bhopal")]     # AND  → use &, wrap in ()
df[(df["city"] == "Pune") | (df["age"] < 26)]       # OR   → use |
df[df["city"].isin(["Delhi", "Pune"])]
df.query("age > 28 and city == 'Bhopal'")           # readable alternative

	.loc	.iloc
Selects by	Labels	Integer positions
Slice end	Inclusive	Exclusive
Example	df.loc[1, "age"]	df.iloc[1, 1]

⚠️ Use &, |, ~ for combining conditions, not and, or, not. And always wrap each condition in parentheses.

4.6 Adding, Changing and Removing Columns

df["salary_k"] = df["salary"] / 1000                    # new column from existing ones
df["senior"] = df["age"] >= 30                          # boolean column
df["grade"] = np.where(df["salary"] > 60000, "high", "low")

df = df.rename(columns={"salary": "income"})
df = df.drop(columns=["salary_k"])                      # remove a column
df = df.drop(index=[0])                                 # remove a row by label

4.7 Sorting

df.sort_values("age")                                   # ascending
df.sort_values("salary", ascending=False)
df.sort_values(["city", "age"], ascending=[True, False])
df.sort_index()

4.8 Aggregation and GroupBy

GroupBy follows split → apply → combine: split rows into groups, compute something per group, combine the results.

df["salary"].mean()                         # one number

df.groupby("city")["salary"].mean()         # average salary per city

df.groupby("city").agg(
    avg_salary=("salary", "mean"),
    max_age=("age", "max"),
    headcount=("name", "count"),
)

4.9 Combining DataFrames

# Stack vertically / side by side
pd.concat([df1, df2], axis=0, ignore_index=True)      # add rows
pd.concat([df1, df2], axis=1)                         # add columns

# Join on a key column (like SQL JOIN)
pd.merge(customers, orders, on="customer_id", how="left")

how=	Keeps
inner	Only keys present in both tables
left	All rows from the left, matches from the right
right	All rows from the right, matches from the left
outer	All rows from both

4.10 apply: Custom Functions

df["age_group"] = df["age"].apply(lambda a: "young" if a < 30 else "adult")
df["name_length"] = df["name"].apply(len)

⚡ Built-in vectorized operations (df["age"] * 2, np.where, .str methods) are faster than apply. Use apply only when there's no vectorized alternative.

🧠 Why ML cares

Real datasets arrive as CSV files or database tables. Pandas is how you load, inspect, filter, join and reshape that data into the clean feature table that ML models expect.

---

5. Data Cleaning and Manipulation

"Garbage in, garbage out." Data scientists commonly spend the majority of their time cleaning data. Better data usually beats a fancier model.

5.1 A Typical Cleaning Checklist

1. Inspect        → shape, dtypes, head, describe, info
2. Fix types      → numbers stored as text, dates as strings
3. Missing values → find, then drop or fill
4. Duplicates     → find and remove
5. Inconsistent   → typos, spacing, mixed capitalisation, units
6. Outliers       → detect, then decide what to do
7. Encode & scale → turn data into model-ready numbers
8. Validate       → re-check everything

5.2 Handling Missing Values

Missing data appears as NaN (Not a Number) or None.

df.isna().sum()                  # count of missing values per column
df.isna().mean() * 100           # percentage missing per column
df[df["age"].isna()]             # rows where age is missing

Options

Strategy	Code	Good when
Drop rows	df.dropna()	Few rows are missing and data is plentiful
Drop columns	df.drop(columns=["col"])	A column is mostly empty
Fill with a constant	df["x"].fillna(0)	A sensible default exists (e.g., 0 for "no purchases")
Fill with mean / median	df["age"].fillna(df["age"].median())	Numeric columns (median is safer with outliers)
Fill with mode	df["city"].fillna(df["city"].mode()[0])	Categorical columns
Forward / backward fill	df["x"].ffill(), df["x"].bfill()	Time-ordered data
Interpolate	df["x"].interpolate()	Smoothly varying numeric series

df = df.dropna(subset=["salary"])                         # drop rows missing salary only
df["age"] = df["age"].fillna(df["age"].median())
df = df.dropna(thresh=3)                                  # keep rows with at least 3 non-null values

💡 Always ask why data is missing. Sometimes the missingness itself carries information. (A common trick is adding an is_missing indicator column before filling.)

5.3 Removing Duplicates

df.duplicated().sum()                          # how many duplicate rows
df = df.drop_duplicates()
df = df.drop_duplicates(subset=["email"], keep="first")

5.4 Fixing Data Types

df["age"] = pd.to_numeric(df["age"], errors="coerce")      # bad values → NaN
df["join_date"] = pd.to_datetime(df["join_date"])
df["city"] = df["city"].astype("category")                 # saves memory
df["price"] = df["price"].astype(float)

5.5 Cleaning Text Columns

df["city"] = df["city"].str.strip()                       # remove extra spaces
df["city"] = df["city"].str.lower()                       # consistent case
df["city"] = df["city"].str.replace("-", " ")
df["name"].str.contains("As")                             # boolean mask
df["email"].str.split("@").str[1]                         # extract the domain
df["price"] = df["price"].str.replace("$", "", regex=False).astype(float)
df["city"] = df["city"].replace({"bhopal ": "bhopal", "delhi": "new delhi"})

5.6 Working with Dates

df["join_date"] = pd.to_datetime(df["join_date"])
df["year"]  = df["join_date"].dt.year
df["month"] = df["join_date"].dt.month
df["weekday"] = df["join_date"].dt.day_name()
df["days_since"] = (pd.Timestamp("2026-01-01") - df["join_date"]).dt.days

Splitting a date into parts (year, month, weekday) creates useful features.

5.7 Detecting and Handling Outliers

Outliers are unusual values far from the rest. They can be errors or genuine extremes.

The IQR rule

Q1 = df["salary"].quantile(0.25)
Q3 = df["salary"].quantile(0.75)
IQR = Q3 - Q1
lower, upper = Q1 - 1.5 * IQR, Q3 + 1.5 * IQR

outliers = df[(df["salary"] < lower) | (df["salary"] > upper)]

The z-score rule

z = (df["salary"] - df["salary"].mean()) / df["salary"].std()
df[z.abs() > 3]                  # more than 3 standard deviations from the mean

What to do with them

• Remove them if they are clear errors
• Cap / winsorize: df["salary"].clip(lower, upper)
• Transform (e.g., log) to reduce skew: np.log1p(df["salary"])
• Keep them if they're real and meaningful (e.g., fraud cases)

5.8 Encoding Categorical Variables

Models need numbers, so categories must be converted.

# One-hot encoding: one 0/1 column per category (for categories with no natural order)
pd.get_dummies(df, columns=["city"], drop_first=True)

# Label / ordinal encoding: integers for categories WITH a natural order
order = {"low": 0, "medium": 1, "high": 2}
df["level_num"] = df["level"].map(order)

Encoding	Use when	Watch out
One-hot	Categories have no order (city, colour)	Many categories → many columns
Ordinal	Categories have a meaningful order (low/med/high)	Don't use for unordered categories, since the model would assume false distances

drop_first=True removes one redundant column to avoid perfect redundancy between the dummy columns.

5.9 Scaling and Normalising Numeric Columns

Features on very different scales (age: 20–60, salary: 20,000–200,000) can hurt many algorithms.

# Min-max scaling → range [0, 1]
df["age_mm"] = (df["age"] - df["age"].min()) / (df["age"].max() - df["age"].min())

# Standardisation (z-score) → mean 0, std 1
df["age_z"] = (df["age"] - df["age"].mean()) / df["age"].std()

(Scikit-learn offers MinMaxScaler and StandardScaler that do this properly. See Section 8.)

5.10 Reshaping and Summarising

# Binning a numeric column into ranges
df["age_band"] = pd.cut(df["age"], bins=[0, 25, 35, 100], labels=["<25", "25-35", "35+"])
pd.qcut(df["salary"], q=4)                        # equal-sized groups (quartiles)

# Pivot table: Excel-style summary
pd.pivot_table(df, values="salary", index="city", columns="senior", aggfunc="mean")

# Crosstab: counts of category combinations
pd.crosstab(df["city"], df["senior"])

# Wide ↔ long format
pd.melt(df, id_vars=["name"], value_vars=["age", "salary"])

# Correlations
df.corr(numeric_only=True)

5.11 Method Chaining: Readable Pipelines

clean = (
    pd.read_csv("raw.csv")
      .drop_duplicates()
      .rename(columns=str.lower)
      .assign(age=lambda d: d["age"].fillna(d["age"].median()))
      .query("salary > 0")
      .reset_index(drop=True)
)

⚠️ Avoid chained assignment like df[df["age"] > 30]["salary"] = 0. It may silently fail or change a copy. Use df.loc[df["age"] > 30, "salary"] = 0 instead. Use df.copy() when you need an independent copy.

🧠 Why ML cares

Clean, consistent, numeric data is a prerequisite for training. Mistakes made here (such as leaking information or mishandling missing values) quietly damage every later step.

---

6. Matplotlib and Seaborn Visualization

"A picture is worth a thousand rows." Visualization helps you understand data, spot problems, and communicate results.

import matplotlib.pyplot as plt
import seaborn as sns

	Matplotlib	Seaborn
Role	The foundation: full control over every detail	A high-level layer on top of Matplotlib
Strength	Highly customisable	Beautiful statistical plots in one line
Works with	Arrays, lists, DataFrames	DataFrames (especially "tidy" ones)
Typical use	Custom figures, fine-tuning	Quick exploration, distributions, relationships

They work together. Seaborn draws, Matplotlib fine-tunes.

6.1 Matplotlib Basics

Anatomy of a figure

Figure   → the whole canvas (the "window" or image)
 └─ Axes → one individual plot (with x-axis, y-axis, title, legend, ...)

The simple way

x = [1, 2, 3, 4, 5]
y = [2, 4, 5, 4, 6]

plt.plot(x, y)
plt.title("Simple Line")
plt.xlabel("x")
plt.ylabel("y")
plt.show()

The recommended way: object-oriented (fig, ax)

fig, ax = plt.subplots(figsize=(6, 4))
ax.plot(x, y, color="tab:blue", linestyle="--", marker="o", label="trend")
ax.set_title("Simple Line")
ax.set_xlabel("x")
ax.set_ylabel("y")
ax.legend()
ax.grid(True, alpha=0.3)
fig.tight_layout()
fig.savefig("plot.png", dpi=150)
plt.show()

6.2 Common Matplotlib Plot Types

fig, axes = plt.subplots(2, 2, figsize=(10, 8))      # a 2×2 grid of plots

axes[0, 0].plot(x, y)                                # Line plot (trends over time)
axes[0, 0].set_title("Line")

axes[0, 1].scatter(df["age"], df["salary"])          # Scatter (relationship of 2 variables)
axes[0, 1].set_title("Scatter")

axes[1, 0].bar(["A", "B", "C"], [10, 24, 17])        # Bar (compare categories)
axes[1, 0].set_title("Bar")

axes[1, 1].hist(df["age"], bins=10, edgecolor="black")   # Histogram (distribution)
axes[1, 1].set_title("Histogram")

fig.tight_layout()
plt.show()

Plot	Function	Best for
Line	ax.plot	Trends over time, training loss curves
Scatter	ax.scatter	Relationship between two numeric variables
Bar	ax.bar / ax.barh	Comparing categories
Histogram	ax.hist	Distribution of one numeric variable
Box plot	ax.boxplot	Spread, median and outliers
Pie	ax.pie	Parts of a whole (use sparingly)
Image	ax.imshow	Images, matrices, heatmaps

Useful customisations

ax.set_xlim(a, b), ax.set_xticks(...), ax.tick_params(axis="x", rotation=45), ax.axhline(0) (reference line), ax.annotate(...), plt.style.use("seaborn-v0_8").

6.3 Seaborn Basics

Seaborn works directly with DataFrames, using column names.

sns.set_theme(style="whitegrid")          # nicer default look

Distribution plots (one variable)

sns.histplot(data=df, x="age", bins=15, kde=True)     # histogram + smooth density curve
sns.kdeplot(data=df, x="salary", fill=True)           # smoothed density
sns.boxplot(data=df, x="city", y="salary")            # median, quartiles, outliers per group
sns.violinplot(data=df, x="city", y="salary")         # box plot + distribution shape

Relationship plots (two variables)

sns.scatterplot(data=df, x="age", y="salary", hue="city", size="age")
sns.lineplot(data=df, x="year", y="sales")
sns.regplot(data=df, x="age", y="salary")             # scatter + fitted line

hue colours points by a category, so you can show a third variable.

Categorical plots

sns.countplot(data=df, x="city")                      # count of each category
sns.barplot(data=df, x="city", y="salary")            # mean per category (with error bars)

Correlation heatmap: a must-do in ML exploration

corr = df.corr(numeric_only=True)
sns.heatmap(corr, annot=True, cmap="coolwarm", vmin=-1, vmax=1, fmt=".2f")
plt.title("Feature Correlations")
plt.show()

Shows at a glance which features move together and which relate to the target.

Pairwise relationships

sns.pairplot(df, hue="city")                          # grid of scatter plots + distributions

Subplots with Seaborn

fig, axes = plt.subplots(1, 2, figsize=(10, 4))
sns.histplot(df["age"], ax=axes[0])                   # pass ax= to place a plot in a subplot
sns.boxplot(x=df["salary"], ax=axes[1])
plt.show()

6.4 Which Chart Should I Use?

Question	Chart
How is one numeric variable distributed?	Histogram, KDE, box plot
How do two numeric variables relate?	Scatter plot (+ regression line)
How do many numeric variables relate?	Correlation heatmap, pair plot
How do values compare across categories?	Bar plot, box plot
How often does each category occur?	Count plot
How does something change over time?	Line plot
Spotting outliers?	Box plot, scatter plot

6.5 ML-Specific Plot Examples

# Training curves
plt.plot(train_losses, label="train")
plt.plot(val_losses, label="validation")
plt.xlabel("epoch"); plt.ylabel("loss"); plt.legend(); plt.show()

# Class balance of the target
sns.countplot(x=y)

# Confusion matrix as a heatmap
sns.heatmap(cm, annot=True, fmt="d", cmap="Blues")

6.6 Good Visualization Habits

• ✅ Always label axes and add a title (include units)
• ✅ Choose the chart type that matches the question
• ✅ Keep it simple: avoid clutter and 3-D effects
• ✅ Use colour meaningfully (and consider colour-blind-friendly palettes)
• ✅ Start bar charts at zero
• ❌ Don't use pie charts for many categories

🧠 Why ML cares

Plots reveal skewed distributions, outliers, class imbalance, correlated features and data errors before modelling, and show how training is going after it.

---

7. SciPy for Scientific Computing

SciPy builds on NumPy and adds a large collection of scientific and numerical algorithms. Where NumPy gives you arrays, SciPy gives you ready-made tools to do science with them.

import scipy
from scipy import stats, optimize, linalg, integrate, sparse, interpolate, spatial

7.1 What's Inside

Submodule	What it provides
scipy.stats	Probability distributions, statistical tests, descriptive statistics
scipy.optimize	Minimisation, curve fitting, root finding
scipy.linalg	Linear algebra routines (more complete than numpy.linalg)
scipy.sparse	Memory-efficient sparse matrices
scipy.integrate	Numerical integration and differential equations
scipy.interpolate	Interpolation (estimating values between known points)
scipy.spatial	Distances, nearest neighbours, spatial structures
scipy.special	Special functions (e.g., sigmoid via expit, gamma)
scipy.signal, scipy.fft	Signal processing and Fourier transforms

7.2 scipy.stats: Distributions and Statistical Tests

Working with distributions

from scipy import stats

norm = stats.norm(loc=0, scale=1)       # standard normal: mean 0, std 1

norm.pdf(0)                    # density at 0       → ≈ 0.3989
norm.cdf(1.96)                 # P(X ≤ 1.96)        → ≈ 0.975
norm.ppf(0.975)                # inverse CDF        → ≈ 1.96   (the "critical value")
norm.rvs(size=5, random_state=42)    # draw random samples

Method	Meaning
pdf / pmf	Density (continuous) / probability (discrete) at a value
cdf	Cumulative probability P(X ≤ x)
ppf	Percent-point function (inverse of CDF)
sf	Survival function = 1 − CDF
rvs	Random variates (samples)

Other distributions: stats.binom, stats.poisson, stats.uniform, stats.expon, stats.t, stats.chi2.

Statistical tests

a = [23, 25, 28, 30, 31]
b = [30, 33, 35, 38, 40]

t_stat, p_value = stats.ttest_ind(a, b)         # compare two group means
# small p-value (< 0.05) → evidence that the means differ

stats.ttest_1samp(a, popmean=25)                # compare one sample to a known mean
stats.chi2_contingency(table)                   # relationship between categorical variables
stats.shapiro(a)                                # test for normality
stats.f_oneway(g1, g2, g3)                      # ANOVA across 3+ groups

Correlation and descriptive statistics

r, p = stats.pearsonr(x, y)                     # linear correlation + p-value
rho, p = stats.spearmanr(x, y)                  # rank (monotonic) correlation
stats.describe(a)                               # count, min/max, mean, variance, skew, kurtosis
stats.zscore(a)                                 # standardise values
stats.skew(a), stats.kurtosis(a)

7.3 scipy.optimize: Finding Minima and Fitting Curves

Minimising a function

from scipy.optimize import minimize

def f(p):                            # f(x, y) = (x − 3)² + (y + 1)²
    x, y = p
    return (x - 3) ** 2 + (y + 1) ** 2

result = minimize(f, x0=[0, 0])      # x0 = starting guess
result.x                             # ≈ [ 3., -1.]   ← the minimiser
result.fun                           # ≈ 0            ← the minimum value
result.success                       # True if it converged

minimize can also use constraints and bounds, and you can pick a method (method="BFGS", "L-BFGS-B", "SLSQP").

Curve fitting

from scipy.optimize import curve_fit
import numpy as np

def model(x, a, b):
    return a * np.exp(b * x)

params, covariance = curve_fit(model, x_data, y_data)
a_fit, b_fit = params

Root finding

from scipy.optimize import root_scalar
root_scalar(lambda x: x**2 - 4, bracket=[0, 5]).root      # 2.0

7.4 scipy.linalg

from scipy import linalg
A = np.array([[4., 1.], [2., 3.]])

linalg.inv(A)
linalg.det(A)
linalg.eig(A)
linalg.svd(A)
linalg.solve(A, np.array([1., 2.]))

(Includes many extra decompositions such as LU, QR and Cholesky.)

7.5 scipy.sparse: Matrices Full of Zeros

Many ML datasets (e.g., text with thousands of word-columns) are mostly zeros. A sparse matrix stores only the non-zero entries, saving huge amounts of memory.

from scipy import sparse

dense = np.array([[0, 0, 3],
                  [4, 0, 0],
                  [0, 0, 0]])

sp = sparse.csr_matrix(dense)        # Compressed Sparse Row format
sp.nnz                               # 2 → number of stored (non-zero) values
sp.toarray()                         # convert back to a normal array

Format	Good for
csr_matrix	Fast row slicing and matrix-vector products
csc_matrix	Fast column slicing
coo_matrix	Easy construction from (row, col, value) triplets

Scikit-learn accepts sparse matrices directly in many places.

7.6 scipy.integrate

from scipy.integrate import quad

value, error = quad(lambda x: x**2, 0, 1)      # ∫₀¹ x² dx
value                                           # ≈ 0.3333

Also includes solve_ivp for differential equations.

7.7 scipy.interpolate

from scipy.interpolate import interp1d

x = [0, 1, 2, 3]
y = [0, 1, 4, 9]
f = interp1d(x, y, kind="linear")
f(1.5)                                          # estimated value at 1.5 → 2.5

7.8 scipy.spatial: Distances

from scipy.spatial.distance import cdist, euclidean, cosine

euclidean([0, 0], [3, 4])                       # 5.0
cosine([1, 0], [0, 1])                          # 1.0 (cosine DISTANCE = 1 − similarity)

A = np.array([[0, 0], [1, 1]])
B = np.array([[3, 4], [0, 1]])
cdist(A, B, metric="euclidean")                 # all pairwise distances (2×2 matrix)

Distance calculations are central to similarity-based methods.

NumPy vs SciPy at a glance

	NumPy	SciPy
Core	Arrays and basic math	Algorithms built on arrays
Think of it as	The foundation	The toolbox
Examples	np.mean, np.dot	stats.ttest_ind, optimize.minimize

🧠 Why ML cares

Statistical testing, distribution fitting, numerical optimisation, sparse data handling and distance calculations are everyday ML needs, and SciPy provides all of them (and Scikit-learn uses SciPy internally).

---

8. Scikit-learn Fundamentals

Scikit-learn (sklearn) is the most widely used library for classical machine learning. It offers a clean, consistent interface for dozens of algorithms plus all the tools around them: preprocessing, splitting data, evaluation and tuning.

pip install scikit-learn        # install name
import sklearn                  # import name

8.1 The Consistent API: the Heart of Scikit-learn

Almost everything in Scikit-learn is an estimator with the same few methods. Learn the pattern once and you can use any model.

Object type	Main methods	Purpose	Examples
Estimator	fit(X, y)	Learn from data	Every model and transformer
Predictor (model)	predict(X), score(X, y)	Make predictions	LinearRegression, LogisticRegression, RandomForestClassifier
Transformer	transform(X), fit_transform(X)	Modify data	StandardScaler, SimpleImputer, OneHotEncoder

model = SomeModel(hyperparameter=value)     # 1. create (set hyperparameters)
model.fit(X_train, y_train)                 # 2. learn from training data
predictions = model.predict(X_test)         # 3. predict on new data

Conventions to remember

• X = feature matrix with shape (n_samples, n_features)
• y = target vector with shape (n_samples,)
• Hyperparameters are set in the constructor; learned values end in an underscore (e.g., model.coef_, model.intercept_)

8.2 Datasets

from sklearn.datasets import load_iris, make_classification

X, y = load_iris(return_X_y=True)          # small built-in practice dataset
X.shape                                    # (150, 4)

X, y = make_classification(n_samples=500, n_features=10, random_state=42)   # synthetic data

8.3 Splitting the Data

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(
    X, y,
    test_size=0.2,          # 20% for testing
    random_state=42,        # reproducible split
    stratify=y,             # keep class proportions equal (classification)
)

(Why split? To test on data the model has never seen. See the ML Fundamentals notes on generalisation.)

8.4 A Complete First Example

from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score

X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

model = LogisticRegression(max_iter=200)
model.fit(X_train, y_train)

y_pred = model.predict(X_test)
print("Accuracy:", accuracy_score(y_test, y_pred))

Swap LogisticRegression for another classifier and the other lines stay the same. That's the power of the consistent API.

8.5 Preprocessing with Transformers

from sklearn.preprocessing import StandardScaler, MinMaxScaler, OneHotEncoder, LabelEncoder
from sklearn.impute import SimpleImputer

Transformer	What it does
StandardScaler	Rescale each feature to mean 0, std 1
MinMaxScaler	Rescale each feature to the range [0, 1]
SimpleImputer	Fill missing values (mean, median, most frequent, constant)
OneHotEncoder	One-hot encode categorical features
OrdinalEncoder	Integer-encode ordered categories
LabelEncoder	Encode the target labels as integers

scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)    # LEARN mean/std from training data, then apply
X_test_scaled  = scaler.transform(X_test)         # APPLY the same values to test data (no fit!)

🚨 Data leakage warning: call fit / fit_transform on the training data only. Applying transform to test data with the training statistics keeps test information out of training. Fitting a scaler on the full dataset before splitting is a classic mistake.

imputer = SimpleImputer(strategy="median")        # also "mean", "most_frequent", "constant"
X_train_filled = imputer.fit_transform(X_train)

8.6 Pipelines: Chain Steps Safely

A Pipeline bundles preprocessing and a model into one object. It prevents data leakage and keeps code tidy.

from sklearn.pipeline import Pipeline

pipe = Pipeline([
    ("imputer", SimpleImputer(strategy="median")),
    ("scaler",  StandardScaler()),
    ("model",   LogisticRegression(max_iter=200)),
])

pipe.fit(X_train, y_train)          # runs every step in order on the training data
pipe.predict(X_test)                # applies the same steps, then predicts
pipe.score(X_test, y_test)

Different columns, different treatment: ColumnTransformer

from sklearn.compose import ColumnTransformer

numeric_cols = ["age", "salary"]
categorical_cols = ["city"]

preprocess = ColumnTransformer([
    ("num", Pipeline([
        ("impute", SimpleImputer(strategy="median")),
        ("scale",  StandardScaler()),
    ]), numeric_cols),
    ("cat", OneHotEncoder(handle_unknown="ignore"), categorical_cols),
])

full_pipe = Pipeline([
    ("prep",  preprocess),
    ("model", LogisticRegression(max_iter=500)),
])

full_pipe.fit(df_train[numeric_cols + categorical_cols], y_train)

This works directly with Pandas DataFrames, with each column group processed appropriately.

8.7 Model Evaluation

from sklearn.metrics import (
    accuracy_score, precision_score, recall_score, f1_score,
    confusion_matrix, classification_report,
    mean_absolute_error, mean_squared_error, r2_score,
)

Classification

accuracy_score(y_test, y_pred)
confusion_matrix(y_test, y_pred)
print(classification_report(y_test, y_pred))      # precision, recall, F1 for every class

Regression

mean_absolute_error(y_test, y_pred)               # MAE
mean_squared_error(y_test, y_pred)                # MSE
mean_squared_error(y_test, y_pred) ** 0.5         # RMSE
r2_score(y_test, y_pred)                          # R²: 1.0 is perfect

Task	Common metrics
Classification	Accuracy, Precision, Recall, F1, ROC-AUC, Confusion matrix
Regression	MAE, MSE / RMSE, R²

8.8 Cross-Validation: A More Reliable Estimate

A single train/test split can be lucky or unlucky. k-fold cross-validation splits the training data into k parts, trains on k − 1 of them and validates on the remaining one, rotating until every part has been the validation set.

Fold 1: [VAL][train][train][train][train]
Fold 2: [train][VAL][train][train][train]
Fold 3: [train][train][VAL][train][train]
  ...and so on, then average the k scores.

from sklearn.model_selection import cross_val_score

scores = cross_val_score(pipe, X_train, y_train, cv=5, scoring="accuracy")
print(scores.mean(), scores.std())          # average performance and its variability

Others: StratifiedKFold (keeps class balance in each fold), cross_validate (several metrics at once).

8.9 Hyperparameter Tuning

Hyperparameters are settings you choose before training (e.g., regularisation strength). Searching over them systematically:

from sklearn.model_selection import GridSearchCV, RandomizedSearchCV

param_grid = {"model__C": [0.01, 0.1, 1, 10]}       # 'stepname__param' for pipelines

grid = GridSearchCV(pipe, param_grid, cv=5, scoring="accuracy")
grid.fit(X_train, y_train)

grid.best_params_                  # best settings found
grid.best_score_                   # best cross-validated score
best_model = grid.best_estimator_  # refit on all training data with the best settings
best_model.score(X_test, y_test)   # final check on the untouched test set

Method	How it works
GridSearchCV	Tries every combination (thorough, can be slow)
RandomizedSearchCV	Tries random combinations (faster for big search spaces)

8.10 A Tour of Model Families Available

Just recognising where to find things (the details belong to later topics):

Module	Examples
sklearn.linear_model	LinearRegression, LogisticRegression, Ridge, Lasso
sklearn.neighbors	KNeighborsClassifier
sklearn.tree	DecisionTreeClassifier, DecisionTreeRegressor
sklearn.ensemble	RandomForestClassifier, GradientBoostingClassifier
sklearn.svm	SVC, SVR
sklearn.naive_bayes	GaussianNB
sklearn.cluster	KMeans, DBSCAN
sklearn.decomposition	PCA

8.11 Saving and Loading Models

import joblib

joblib.dump(best_model, "model.joblib")        # save
loaded = joblib.load("model.joblib")           # load later
loaded.predict(X_new)

Save the whole pipeline so preprocessing is applied identically in future.

8.12 Handy Tips

• Always set random_state for reproducibility.
• Check model.get_params() to see all hyperparameters.
• Use help(Model) or the official docs to see what each parameter means.
• Start with a simple baseline model before trying complex ones.

🧠 Why ML cares

Scikit-learn turns the whole workflow: split → preprocess → train → evaluate → tune → save, into a few readable lines with a consistent interface.

---

9. Jupyter Notebook and Google Colab

9.1 What Are They?

Both let you write code, results, text, equations and charts in one document, which makes them ideal for experimentation, learning and sharing.

	Jupyter Notebook	Google Colab
Runs	On your own computer (or a server)	In the cloud (Google's servers)
Setup	Install Python + Jupyter	None: just open in the browser with a Google account
Cost	Free	Free tier (paid plans exist for more resources)
Hardware	Your machine's CPU/GPU	Free access to GPU/TPU (limited and not guaranteed)
Packages	You install and manage them	Many ML libraries pre-installed
Files	Local file system	Temporary cloud storage; mount Google Drive to persist files
Sessions	Run as long as you like	Sessions time out after inactivity or a maximum duration
Sharing	Share the .ipynb file	Share a link, like Google Docs
Offline	✅ Works offline	❌ Needs internet

Both use the .ipynb file format (IPython Notebook).

9.2 Starting Jupyter Notebook

pip install notebook            # or: pip install jupyterlab
jupyter notebook                # opens in your browser
# or
jupyter lab                     # the newer, more full-featured interface

In VS Code, you can also open .ipynb files directly.

9.3 Core Concepts

Cells: the building blocks

Cell type	Contains	Run result
Code	Python code	Output shown beneath the cell
Markdown	Formatted text (headings, lists, links, LaTeX equations like $E = mc^2$)	Rendered text
Raw	Unformatted text	Not executed

The kernel

The kernel is the Python engine running behind the notebook. It remembers variables between cells.

• Cells can be run in any order, which is powerful but can cause confusion. If you run cells out of order, you may see results that don't match the code as written top to bottom.
• Restart the kernel to clear all variables and start fresh.
• "Restart & Run All" is the best way to confirm your notebook works from top to bottom.

Cell [1]:  x = 10
Cell [2]:  y = x * 2          ← depends on x existing in the kernel's memory
Cell [3]:  print(y)           ← prints 20

Output of the last line

The last expression in a cell is displayed automatically, with no print needed:

df.head()          # shows a neatly formatted table

9.4 Essential Keyboard Shortcuts (Jupyter)

Two modes: Edit mode (typing in a cell, green border) and Command mode (managing cells, blue border). Press Esc for command mode and Enter for edit mode.

Shortcut	Action
Shift + Enter	Run cell and move to the next
Ctrl + Enter	Run cell and stay
Esc / Enter	Command mode / Edit mode
A / B (command mode)	Insert cell Above / Below
D, D (press D twice)	Delete cell
M / Y	Change cell to Markdown / code (Y)
Z	Undo cell deletion
Tab	Autocomplete
Shift + Tab	Show a function's documentation

(Colab uses similar ideas but with different keys, such as Ctrl + M followed by a letter. Check Tools → Keyboard shortcuts.)

9.5 Magic Commands

Special commands that start with % (one line) or %% (whole cell).

Command	Purpose
%matplotlib inline	Show plots inside the notebook (often the default)
%timeit expr	Time a short expression precisely
%%time	Time the whole cell
%who / %whos	List variables in memory
%run script.py	Run an external Python file
%load_ext autoreload %autoreload 2	Auto-reload edited modules
!command	Run a shell command, e.g. !ls, !pip install seaborn
?name or help(name)	Show documentation

%timeit sum(range(1000))
!pip list

9.6 Google Colab Specifics

Getting started

1. Go to colab.research.google.com and sign in with a Google account.
2. Create a New notebook or open one from Google Drive or GitHub.

Using a GPU / TPU

Runtime → Change runtime type → choose a hardware accelerator. Check it with:

!nvidia-smi          # shows GPU info (if a GPU is assigned)

Installing packages

!pip install some-package        # installs only for the current session

Colab sessions reset when they end, so reinstall in each new session (put install commands in the first cell).

Working with files

Upload a file (temporary):

from google.colab import files
uploaded = files.upload()

Mount Google Drive (so files persist):

from google.colab import drive
drive.mount("/content/drive")

df = pd.read_csv("/content/drive/MyDrive/data/titanic.csv")

Download a file:

files.download("results.csv")

⚠️ Files saved in Colab's local storage (/content/) are deleted when the session ends. Save important outputs to Drive.

Colab niceties

• Text cells support Markdown and LaTeX
• Forms and Table of Contents for tidy notebooks
• Saves automatically to Google Drive, with version history
• Easy sharing and commenting

9.7 Best Practices for Notebooks

• ✅ Give the notebook a clear structure: title, sections with Markdown headings, short explanations
• ✅ Put imports and installs in the first cells
• ✅ Keep cells small and focused
• ✅ Set random seeds for reproducibility
• ✅ Restart & Run All before sharing to confirm it works top-to-bottom
• ✅ Move reusable code into functions (or .py modules)
• ❌ Avoid hidden state: don't rely on cells you ran once and deleted
• ❌ Don't commit large outputs or secrets (API keys, passwords) to shared notebooks

Use notebooks for	Prefer .py scripts for
Exploration, visualization, teaching, reporting	Production code, automation, reusable modules

🧠 Why ML cares

ML is experimental: you try something, look at the result, adjust. Notebooks give instant feedback with code, plots and notes together, and Colab removes the setup and hardware barrier.

---

10. Virtual Environments and Package Management

10.1 The Problem They Solve

Different projects need different versions of libraries:

Project A needs  scikit-learn 1.2   ┐   Installing both globally
Project B needs  scikit-learn 1.5   ┘   → they CONFLICT 💥

A virtual environment is an isolated folder with its own Python interpreter and its own set of installed packages. Each project gets its own, so nothing clashes.

 System Python (don't pollute it!)
 ├── project_A/  → .venv with scikit-learn 1.2
 └── project_B/  → .venv with scikit-learn 1.5

Benefits

• ✅ No version conflicts between projects
• ✅ Reproducibility: others can recreate your exact setup
• ✅ Keeps your system Python clean
• ✅ Safe to experiment (just delete the environment if it breaks)

10.2 Option A: venv (built into Python)

Create

python -m venv .venv            # creates a folder named .venv

(On some systems, use python3 instead of python.)

Activate

Operating system	Command
Windows (Command Prompt)	.venv\Scripts\activate
Windows (PowerShell)	.venv\Scripts\Activate.ps1
macOS / Linux	source .venv/bin/activate

When active, your terminal prompt shows (.venv).

Deactivate

deactivate

Typical workflow

mkdir my_ml_project && cd my_ml_project
python -m venv .venv
source .venv/bin/activate                    # (Windows: .venv\Scripts\activate)
pip install numpy pandas matplotlib seaborn scikit-learn jupyter
pip freeze > requirements.txt                # record exact versions

10.3 Package Management with pip

pip is Python's standard package installer (it downloads from PyPI, the Python Package Index).

Task	Command
Install a package	pip install pandas
Install a specific version	pip install pandas==2.2.2
Install with a version range	pip install "pandas>=2.0,<3.0"
Upgrade a package	pip install --upgrade pandas
Uninstall	pip uninstall pandas
List installed packages	pip list
Show package details	pip show pandas
Save the environment	pip freeze > requirements.txt
Recreate an environment	pip install -r requirements.txt
Check for conflicts	pip check

requirements.txt: the shareable recipe

numpy==1.26.4
pandas==2.2.2
scikit-learn==1.5.0
matplotlib==3.9.0
seaborn==0.13.2

Anyone can run pip install -r requirements.txt in a fresh environment to reproduce your setup.

Install name ≠ import name

pip install ...	import ...
scikit-learn	sklearn
opencv-python	cv2
pillow	PIL
pyyaml	yaml
numpy, pandas, matplotlib, seaborn, scipy	same name

10.4 Option B: Conda (Anaconda / Miniconda)

Conda is both an environment manager and a package manager. It can install non-Python dependencies too and lets you choose the Python version per environment, so it's popular in data science.

• Anaconda: big distribution with many data science packages pre-installed.
• Miniconda: minimal installer (just conda + Python). You add what you need.

conda create -n mlenv python=3.11           # create an environment named "mlenv"
conda activate mlenv                        # activate
conda install numpy pandas scikit-learn     # install packages
conda install -c conda-forge seaborn        # install from a specific channel
conda list                                  # list packages
conda env list                              # list all environments
conda deactivate                            # leave the environment
conda remove -n mlenv --all                 # delete the environment

conda env export > environment.yml          # save the environment
conda env create -f environment.yml         # recreate it elsewhere

venv + pip vs Conda

	venv + pip	Conda
Comes with	Python itself	Needs installing separately
Installs from	PyPI	Conda channels (and can also use pip)
Python version choice	Uses the Python that created it	Pick any version per environment
Non-Python dependencies	❌	✅
Size	Light	Heavier
Best for	Simple, lightweight projects	Data science with complex dependencies

⚠️ If you mix conda install and pip install in one environment, install conda packages first, then use pip for the rest, to avoid conflicts.

10.5 Using Your Environment in Jupyter

A notebook only sees packages from the environment its kernel uses. To make your environment appear as a kernel:

# inside the activated environment
pip install ipykernel
python -m ipykernel install --user --name=mlenv --display-name "Python (mlenv)"

Then in Jupyter choose Kernel → Change kernel → Python (mlenv).

(In a notebook cell, %pip install package installs into the running kernel's environment, which is safer than !pip install.)

10.6 Best Practices

• 🗂️ One environment per project, created inside (or named after) the project folder
• 📌 Pin versions in requirements.txt for reproducibility
• 🚫 Add .venv/ to .gitignore. Never commit the environment itself, only requirements.txt
• 🔁 Re-generate requirements.txt after installing new packages
• 🧹 Prefer a fresh environment to fix weird dependency problems
• ❌ Avoid installing everything into the global/system Python
• 📝 Document the Python version your project needs

10.7 Troubleshooting Cheat Table

Problem	Likely cause / fix
ModuleNotFoundError after installing	Wrong environment active, or notebook using a different kernel. Check which python (macOS/Linux) or where python (Windows)
pip installs to the wrong Python	Use python -m pip install ... so pip matches the interpreter
PowerShell blocks activation	Run Set-ExecutionPolicy -Scope CurrentUser RemoteSigned, or use Command Prompt
Version conflicts	Create a clean environment and reinstall from requirements.txt
Package installed but import fails	Check the import name vs install name

🧠 Why ML cares

ML libraries evolve quickly and depend on specific versions of each other. Isolated, reproducible environments save hours of "it works on my machine" debugging and let others reproduce your results.

---

11. Quick Revision Cheat Sheet

🐍 Python & OOP

Concept	Remember
Data structures	List [] mutable, Tuple () immutable, Dict {k: v}, Set {} unique
Comprehension	[f(x) for x in items if cond]
Function	def name(args): ... return value; *args, **kwargs, defaults, lambda
Pitfall	Never use a mutable default argument like bucket=[]
Class	Blueprint; __init__(self, ...) sets attributes; methods use self
OOP pillars	Encapsulation, Inheritance, Polymorphism, Abstraction

🔢 NumPy

Concept	Remember
Array	np.array, shape (rows, cols), one dtype
Indexing	a[row, col], a[:, 1], boolean mask a[a > 0]
Vectorization	Operate on whole arrays, avoid Python loops
Broadcasting	Shapes compatible if equal or 1 (from the right)
axis	axis=0 down the rows (one result per column); axis=1 across columns
Matrix multiply	A @ B (vs A * B element-wise)
Reproducibility	np.random.default_rng(seed)

🐼 Pandas

Task	Code
Load	pd.read_csv("f.csv")
Inspect	head(), info(), describe(), shape
Select	df["col"], df.loc[labels], df.iloc[positions], df[cond]
Missing	isna().sum(), dropna(), fillna()
Duplicates	drop_duplicates()
Group	df.groupby("c")["x"].mean()
Join	pd.merge(a, b, on="k", how="left")
Encode	pd.get_dummies(df, columns=[...])

📊 Visualization

Goal	Plot
Distribution	sns.histplot, sns.boxplot
Relationship	sns.scatterplot, sns.pairplot
Correlations	sns.heatmap(df.corr(), annot=True)
Categories	sns.countplot, sns.barplot
Trends	plt.plot / sns.lineplot

🔬 SciPy

Module	Use
stats	Distributions, t-tests, correlation
optimize	minimize, curve_fit
sparse	Memory-efficient matrices with mostly zeros
spatial.distance	Distances (Euclidean, cosine)
linalg, integrate, interpolate	Linear algebra, integrals, interpolation

🤖 Scikit-learn

split → preprocess (fit on TRAIN only) → fit model → predict → evaluate → tune → save

Concept	Remember
API	fit(), predict(), transform()
Split	train_test_split(X, y, test_size=0.2, random_state=42)
Pipeline	Chain preprocessing + model; prevents leakage
Validation	cross_val_score(model, X, y, cv=5)
Tuning	GridSearchCV / RandomizedSearchCV
Save	joblib.dump(model, "m.joblib")

📓 Notebooks & Environments

Topic	Remember
Jupyter vs Colab	Local vs cloud; Colab gives free GPU but temporary storage
Run cell	Shift + Enter; Restart & Run All before sharing
Magics	%timeit, %%time, !pip install
Virtual env	python -m venv .venv → activate → pip install
Reproduce	pip freeze > requirements.txt / pip install -r requirements.txt
Conda	conda create -n env python=3.11 → conda activate env

---

✅ Final Takeaways

1. Python basics and functions are the glue. Learn lists, dicts, comprehensions and functions well.
2. OOP explains how ML libraries work: models are objects with fit and predict methods.
3. NumPy is the engine. Think in arrays and vectorize instead of looping.
4. Pandas loads and shapes real-world tabular data.
5. Cleaning is where most of the effort and most of the model quality comes from.
6. Matplotlib and Seaborn let you see problems and results.
7. SciPy provides the scientific tools: statistics, optimisation, sparse data, distances.
8. Scikit-learn gives one consistent interface for the whole ML workflow. Use pipelines and always fit on training data only.
9. Jupyter and Colab make experimenting fast and shareable.
10. Virtual environments keep projects isolated and reproducible.

---

Friendly, beginner-to-advanced notes on getting data, understanding it, cleaning it, and shaping it so a model can learn from it. Every concept has a plain-English idea, a small example, runnable code, and a note on why it matters.

---

📑 Table of Contents

Part A: Getting to Know Your Data

1. Data Types and Dataset Formats
2. Data Collection and Data Acquisition
3. Structured, Semi-Structured and Unstructured Data
4. Exploratory Data Analysis (EDA)
5. Data Quality Assessment

Part B: Cleaning 6. Handling Missing Values 7. Handling Duplicate Records 8. Outlier Detection and Treatment

Part C: Transforming Features 9. Categorical Data Encoding (Overview) 10. Label Encoding and One-Hot Encoding 11. Ordinal Encoding and Target Encoding 12. Feature Scaling 13. Normalization and Standardization 14. Discretization and Binning 15. Data Transformation

Part D: Preparing Data for Modelling 16. Handling Imbalanced Datasets 17. Sampling Techniques 18. Train, Validation and Test Splits 19. Data Leakage and Prevention 20. Data Pipelines Using Scikit-learn

21. Quick Revision Cheat Sheet

🗺️ The Big Picture

 Collect ─► Understand ─► Clean ─► Transform ─► Split ─► (Model)
   │           │            │          │          │
 sources     EDA &       missing,    encode,    train /
 formats     quality     duplicates, scale,     val /
                         outliers    bin, ...   test
                                                  ▲
                      ALL of this is wrapped in a PIPELINE so that
                      steps are repeatable and free of leakage

🍳 Analogy: Preprocessing is the mise en place of machine learning, the washing, chopping and measuring before cooking. A brilliant chef (model) can't rescue spoiled ingredients (bad data).

Setup used in code examples

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns

---

1. Data Types and Dataset Formats

1.1 Data Types

Knowing a column's type decides how you can summarise it, plot it, and encode it.

                        Data
                   ┌─────┴──────┐
              Numerical      Categorical            (plus: Text, Date/Time, Boolean)
             ┌────┴────┐    ┌─────┴─────┐
         Discrete  Continuous  Nominal   Ordinal

Type	Meaning	Examples	Typical handling
Numerical: Discrete	Countable whole numbers	Number of children, clicks, rooms	Use as-is, scale if needed
Numerical: Continuous	Any value in a range	Height, price, temperature	Scale, check for skew and outliers
Categorical: Nominal	Categories with no order	Colour, city, blood group	One-hot, target encoding
Categorical: Ordinal	Categories with a meaningful order	Low/Medium/High, education level, star rating	Ordinal encoding
Binary	Exactly two values	Yes/No, Spam/Not spam	0/1
Date/Time	Points in time	Order date, timestamp	Extract year, month, weekday, etc.
Text	Free-form language	Reviews, emails	Text-specific processing

⚠️ A number isn't always numerical. A zip_code or customer_id is stored as digits but is really a category (averaging zip codes is meaningless). Always think about meaning, not just storage type.

Levels of measurement (a more formal view)

Level	Order?	Equal gaps?	True zero?	Example
Nominal	❌	❌	❌	Eye colour
Ordinal	✅	❌	❌	Satisfaction rating
Interval	✅	✅	❌	Temperature in °C (0 °C doesn't mean "no heat")
Ratio	✅	✅	✅	Height, weight, income (0 means none)

Why it matters: ratios like "twice as much" only make sense for ratio data.

Data types in pandas

df.dtypes            # int64, float64, object (text/mixed), bool, datetime64[ns], category
df["city"] = df["city"].astype("category")
df["date"] = pd.to_datetime(df["date"])

1.2 Key Dataset Vocabulary

Term	Meaning
Record / sample / row	One observation
Feature / attribute / column	One measured property
Target / label	What we want to predict
Dimensionality	Number of features
Cardinality	Number of distinct values in a column (df["city"].nunique())
Metadata	Data about the data (column descriptions, units, source, collection date)

1.3 Common Dataset Formats

Format	Extension	Structure	Pros	Cons	pandas
CSV	.csv	Plain-text table, comma-separated	Universal, human-readable	No types, large files are slow, quoting/encoding issues	read_csv
TSV	.tsv / .txt	Same, tab-separated	Avoids comma clashes in text	Same as CSV	read_csv(sep="\t")
Excel	.xlsx	Sheets of cells	Familiar, multiple sheets	Slow, messy formatting, size limits	read_excel
JSON	.json	Nested key-value	Flexible, standard for web APIs	Verbose, needs flattening	read_json, json_normalize
JSON Lines	.jsonl	One JSON object per line	Streams well, append-friendly	Not a single valid JSON doc	read_json(lines=True)
XML	.xml	Tagged hierarchy	Self-describing	Verbose	read_xml
Parquet	.parquet	Columnar, compressed, typed	Fast, small, keeps dtypes	Not human-readable	read_parquet
SQL database	tables	Relational tables	Query only what you need, handles big data	Needs a database	read_sql
HDF5	.h5	Hierarchical binary	Large numeric arrays	Less portable	read_hdf
NumPy	.npy / .npz	Binary arrays	Fast array storage	Arrays only	np.load
Images / audio / video	.jpg, .wav, .mp4 …	Binary media	Natural media format	Needs specialised loading	Other libraries
Pickle / joblib	.pkl, .joblib	Python objects	Save any Python object	Python-only, security risk with untrusted files	read_pickle

df = pd.read_csv("data.csv", sep=",", encoding="utf-8", na_values=["?", "NA", "-"])
df = pd.read_csv("big.csv", usecols=["age", "city"], nrows=1000)       # peek at a big file
for chunk in pd.read_csv("huge.csv", chunksize=100_000):               # process big files in pieces
    process(chunk)

df.to_parquet("data.parquet")          # typed, compressed, fast
df.to_csv("clean.csv", index=False)

Choosing a format: CSV for sharing and simplicity, Parquet for large analytical datasets, JSON for nested/web data, SQL for data that lives in a database.

Common loading problems

• Encoding errors: try encoding="latin-1" or "utf-8-sig"
• Wrong delimiter: check sep=
• Numbers read as text because of stray symbols ("$1,200")
• Missing-value markers such as "?", "N/A", -999 (use na_values=)
• Dates not parsed (use parse_dates=["col"])

---

2. Data Collection and Data Acquisition

Data collection is gathering raw data. Data acquisition is the broader process of obtaining it and bringing it into your system, whether by collecting it yourself, buying it, or pulling it from existing sources.

📌 Your model can only be as good as the data you feed it. Poor collection (biased, tiny, or noisy data) can't be fixed later by clever algorithms.

2.1 Where Data Comes From

Source	Description	Examples
Primary data	Collected first-hand for your specific purpose	Surveys, experiments, interviews, A/B tests, sensors
Secondary data	Already collected by someone else	Public datasets, reports, company records
Internal systems	Your organisation's own data	Databases, CRM, transaction logs, web/app logs
Public/open datasets	Free datasets	Kaggle, UCI Machine Learning Repository, government open-data portals, Hugging Face datasets
APIs	Programmatic access to a service's data	Weather, finance, social platforms
Web scraping	Extracting data from web pages	Product prices, news articles
Sensors / IoT	Automatic measurements	Temperature, GPS, accelerometers
Crowdsourcing	People label or collect data	Image labelling platforms
Synthetic data	Artificially generated	Simulated records when real data is scarce or sensitive

2.2 Collection Methods in Practice

Reading from a database

import sqlite3
conn = sqlite3.connect("shop.db")
df = pd.read_sql("SELECT customer_id, amount, date FROM orders WHERE date >= '2025-01-01'", conn)

Filter and select in SQL so you only pull the rows and columns you need.

Calling an API

import requests

response = requests.get(
    "https://api.example.com/v1/items",
    params={"limit": 100, "page": 1},
    headers={"Authorization": "Bearer YOUR_TOKEN"},
    timeout=10,
)
response.raise_for_status()          # raises an error for 4xx/5xx responses
data = response.json()               # parsed JSON
df = pd.json_normalize(data["items"])

APIs usually have rate limits (maximum requests per time), pagination (data returned in pages) and authentication.

Web scraping

from bs4 import BeautifulSoup

html = requests.get("https://example.com/products", timeout=10).text
soup = BeautifulSoup(html, "html.parser")
names = [tag.get_text(strip=True) for tag in soup.select(".product-name")]

Before scraping, check the site's terms of service and robots.txt, and prefer an official API when one exists.

Streaming vs batch acquisition

	Batch	Streaming
Data arrives	In chunks at intervals (daily file)	Continuously, event by event
Example	Nightly export of sales	Live sensor feed, click stream
Suits	Reports, periodic retraining	Real-time monitoring and decisions

2.3 Questions to Ask Before (and While) Collecting

1. Is it relevant? Does it actually relate to what I'm predicting?
2. Is it representative? Does it match the real situations the model will face (all customer types, seasons, regions)?
3. Is it enough? Both rows (samples) and coverage of variation.
4. Is it labelled? Who labels it, how consistently, at what cost?
5. Is it legal and ethical? Consent, privacy, licences, and sensitive attributes.
6. Is it documented? Source, date, units, definitions.

2.4 Pitfalls in Collection

Pitfall	What goes wrong
Sampling bias	Collected data isn't representative (e.g., an online survey misses people without internet)
Survivorship bias	Only successful or surviving cases are recorded
Selection bias	The way you picked data skews it
Measurement error	Faulty sensors or inconsistent definitions
Label noise	Wrong or inconsistent labels
Stale data	The world changed since collection
Privacy violations	Personal data collected without consent or safeguards

2.5 Good Practices

• ✅ Document everything: source, collection date, units, known issues (a "data dictionary")
• ✅ Keep the raw data untouched. Do cleaning in scripts so it's reproducible
• ✅ Version your datasets so experiments can be reproduced
• ✅ Respect privacy and licences: anonymise or remove personal identifiers where appropriate
• ✅ Store data in sensible formats (e.g., Parquet for large tables)

---

3. Structured, Semi-Structured and Unstructured Data

3.1 The Three Kinds

	Structured	Semi-structured	Unstructured
Organisation	Fixed schema: rows and columns	Has tags/keys but flexible, nested structure	No predefined format
Examples	Spreadsheets, SQL tables, CSV	JSON, XML, YAML, log files, HTML	Text documents, images, audio, video, emails
Stored in	Relational databases	Document stores, files	File systems, data lakes
Easy to query?	✅ Very (SQL)	⚠️ Moderate	❌ Needs extra processing
ML-ready?	Nearly	After flattening	After feature extraction
Share of real-world data	Smaller	Medium	Largest

 Structured                 Semi-structured                 Unstructured
 ┌────┬─────┬──────┐        {                                "The delivery was late
 │ id │ age │ city │          "id": 1,                        but the product is
 ├────┼─────┼──────┤          "profile": {"age": 25},         great, thanks!"
 │ 1  │ 25  │ Pune │          "tags": ["a", "b"]
 └────┴─────┴──────┘        }                                 [ an image of pixels ]

3.2 Working with Each Kind

Structured

Ready for tables and SQL. Concentrate on cleaning and encoding.

Semi-structured: flatten it into a table

import json

records = [
    {"id": 1, "user": {"name": "Asha", "age": 25}, "tags": ["new", "vip"]},
    {"id": 2, "user": {"name": "Ravi", "age": 31}, "tags": []},
]

df = pd.json_normalize(records)
#    id  tags  user.name  user.age       ← nested keys become "user.name", "user.age"

df["n_tags"] = df["tags"].apply(len)       # turn list-valued fields into usable numbers
df = df.explode("tags")                    # or: one row per list element

Common work: flatten nested fields, handle missing keys (not every record has the same fields), parse lists into counts or indicator columns.

Unstructured: convert to numbers first

Models need numeric input, so unstructured data must be turned into features:

Data	Typical way to make it numeric
Text	Clean the text, then counts/weights of words (e.g., bag-of-words, TF-IDF) or learned embeddings
Images	Arrays of pixel values (height × width × channels), usually resized and scaled
Audio	Waveform samples or derived features (e.g., spectrograms)
Video	Sequences of image frames (plus audio)

# Image as numbers
from PIL import Image
img = np.array(Image.open("cat.jpg").resize((64, 64)))     # shape (64, 64, 3)
img = img / 255.0                                          # scale pixels to 0–1

💡 The unstructured → numbers step is called feature extraction. Once you have numeric features, most of the later preprocessing in these notes applies.

---

4. Exploratory Data Analysis (EDA)

EDA is getting to know your data before modelling: summarising it, visualising it, and hunting for patterns, problems and surprises. Coined by statistician John Tukey, it's about asking questions and letting the data answer.

4.1 Goals of EDA

• 🔍 Understand structure, types and size
• 🩺 Find problems: missing values, duplicates, outliers, wrong types
• 📈 Understand distributions and relationships
• 🎯 See how features relate to the target
• 💡 Generate ideas for features and the choice of model
• ✅ Check assumptions (and spot data that doesn't match them)

4.2 The EDA Workflow

1. Overview          → size, types, head/tail, memory
2. Quality scan      → missing, duplicates, impossible values
3. Univariate        → each variable on its own
4. Bivariate         → pairs (feature vs feature, feature vs target)
5. Multivariate      → many at once (correlation matrix, pair plots)
6. Target analysis   → distribution, class balance
7. Write findings    → what to clean / transform / engineer

4.3 Step 1: First Look

df.shape                        # (rows, columns)
df.head()                       # first rows
df.sample(5, random_state=1)    # a random peek (better than head for ordered files)
df.info()                       # dtypes, non-null counts, memory
df.describe()                   # numeric summary: count, mean, std, min, quartiles, max
df.describe(include="object")   # categorical summary: count, unique, top, freq
df.nunique()                    # cardinality of every column

Reading describe(): compare mean vs median (50%). A big gap suggests skew or outliers. Check min and max for impossible values (age = −5 or 999).

4.4 Step 2: Univariate Analysis (One Variable)

Numeric

df["income"].agg(["mean", "median", "std", "min", "max", "skew", "kurt"])

sns.histplot(df["income"], kde=True)             # shape of the distribution
sns.boxplot(x=df["income"])                      # median, spread, outliers

Look for	Meaning
Skewness > 0 (long right tail)	Few very large values (income, prices), a candidate for a log transform
Skewness < 0 (long left tail)	Few very small values
**	Skewness
Multiple peaks (multimodal)	Possibly different groups mixed together
Spikes at odd values (0, 999, −1)	Placeholders for missing data

Categorical

df["city"].value_counts()
df["city"].value_counts(normalize=True)          # proportions
sns.countplot(data=df, y="city", order=df["city"].value_counts().index)

Watch for rare categories, typos ("delhi" vs "Delhi "), and very high cardinality.

4.5 Step 3: Bivariate Analysis (Two Variables)

Pair	Tools
Numeric vs Numeric	Scatter plot, correlation (df["a"].corr(df["b"]))
Numeric vs Categorical	Box/violin plot per category, group means: df.groupby("city")["income"].mean()
Categorical vs Categorical	Cross-tab: pd.crosstab(df["city"], df["bought"], normalize="index")

sns.scatterplot(data=df, x="age", y="income", hue="bought")
sns.boxplot(data=df, x="city", y="income")

4.6 Step 4: Multivariate Analysis (Many Variables)

corr = df.corr(numeric_only=True)
sns.heatmap(corr, annot=True, fmt=".2f", cmap="coolwarm", vmin=-1, vmax=1)

sns.pairplot(df[["age", "income", "tenure", "target"]], hue="target")

Use these to find:

• Strongly correlated feature pairs (redundancy)
• Features correlated with the target (potentially useful)
• Suspiciously high correlation with the target (possible leakage, see Section 19)

4.7 Step 5: Analyse the Target

df["target"].value_counts(normalize=True)        # class balance (classification)
sns.histplot(df["price"], kde=True)              # distribution (regression)

Check for class imbalance and for a skewed numeric target.

4.8 Time-Based Data in EDA

df["date"] = pd.to_datetime(df["date"])
df.set_index("date")["sales"].resample("M").sum().plot()      # monthly trend

Look for trends, seasonality, gaps, and sudden shifts.

4.9 EDA Questions Checklist

• How big is the dataset? Is it large enough?
• Are the data types right?
• How much is missing, and where?
• Are there duplicates?
• Are values plausible (ranges, units)?
• How is each variable distributed? Any heavy skew?
• Any outliers? Errors or real?
• Which features relate to the target?
• Are any features redundant?
• Is the target balanced?
• Does anything look "too good to be true"?

4.10 EDA Tips and Traps

• ✅ Do EDA on the training data only, or at least don't let test-set insights drive decisions (see leakage)
• ✅ Record findings as you go: they become your cleaning to-do list
• ✅ Plot, don't just compute. Different datasets can share the same summary statistics yet look completely different
• ❌ Correlation ≠ causation
• ❌ Don't ignore domain knowledge. Ask what values should look like
• ❌ Don't over-interpret patterns in small samples

🧰 Automated profiling tools (such as ydata-profiling) can generate a quick EDA report. They save time but never replace your own judgement.

---

5. Data Quality Assessment

Data quality assessment measures how fit your data is for its purpose before you start fixing it. Think of it as a health check-up that produces a to-do list.

5.1 Dimensions of Data Quality

Dimension	Question	Example problem
Completeness	Is all required data present?	30% of income is empty
Accuracy	Are values correct and truthful?	Age = 250; a wrong label
Consistency	Do values agree across fields and sources?	"NY", "New York", "new york"; end date earlier than start date
Uniqueness	Are records free of duplicates?	The same customer listed three times
Validity	Do values follow allowed formats and ranges?	Email without @; negative quantity
Timeliness	Is the data up to date?	Prices from 3 years ago
Integrity	Do relationships hold?	An order refers to a customer ID that doesn't exist
Representativeness	Does it reflect the real population?	Only one region is covered
Balance	Are classes/groups reasonably covered?	99.5% of the rows are one class

5.2 A Practical Quality Report

def quality_report(df):
    report = pd.DataFrame({
        "dtype": df.dtypes,
        "missing": df.isna().sum(),
        "missing_%": (df.isna().mean() * 100).round(2),
        "n_unique": df.nunique(),
    })
    report["constant"] = report["n_unique"] <= 1          # same value everywhere
    return report

print(quality_report(df))
print("Duplicate rows:", df.duplicated().sum())

5.3 Specific Checks

# 1. Completeness
df.isna().mean().sort_values(ascending=False)

# 2. Uniqueness
df.duplicated().sum()
df["customer_id"].is_unique

# 3. Validity (rules based on domain knowledge)
df[(df["age"] < 0) | (df["age"] > 120)]
df[df["quantity"] <= 0]
df[~df["email"].str.contains("@", na=False)]

# 4. Consistency
df["state"].str.strip().str.lower().value_counts()       # reveals spelling variants
df[df["end_date"] < df["start_date"]]                    # logical contradictions

# 5. Placeholder values pretending to be real
(df == -999).sum()            # also check 0, 9999, "unknown", "?", ""

# 6. Wrong dtype (numbers stored as text)
pd.to_numeric(df["price"], errors="coerce").isna() & df["price"].notna()

# 7. Suspicious columns
df.columns[df.nunique() == 1]                             # constant columns carry no information
df.select_dtypes("object").nunique()                      # very high cardinality, e.g. IDs

# 8. Referential integrity
orders[~orders["customer_id"].isin(customers["customer_id"])]

# 9. Timeliness
df["date"].min(), df["date"].max()

5.4 Also Assess the Labels

For supervised learning, the target's quality matters hugely:

• Are labels correct and consistently defined?
• Do different labelers disagree on the same item?
• Are there contradictory duplicates (identical inputs, different labels)?

# identical features but different labels = label conflict
feature_cols = [c for c in df.columns if c != "label"]
conflicts = df.groupby(feature_cols)["label"].nunique()
conflicts[conflicts > 1]

5.5 Turning Findings into Actions

Finding	Typical action
High % missing in a column	Impute, add a missing-indicator, or drop the column
Duplicates	Remove (after checking they're not legitimate repeats)
Impossible values	Fix, set to missing, or remove
Inconsistent categories	Standardise text (strip, lowercase, map variants)
Constant columns	Drop
ID-like columns	Drop from features
Wrong dtypes	Convert
Heavy imbalance	See Section 16
Possible leakage columns	Investigate and remove

5.6 Good Practice

• Define data quality rules up front, and re-run them whenever new data arrives
• Keep a log of issues and fixes
• Quality isn't a one-time step. Monitor it for incoming data too

---

6. Handling Missing Values

Missing values are the most common data problem, and they matter because most algorithms can't accept NaN as input.

6.1 Why Is Data Missing? (The Three Types)

Understanding why guides what you do.

Type	Meaning	Example	Risk
MCAR, Missing Completely At Random	Missingness has no pattern; pure chance	A sensor randomly drops a reading	Safest. Deleting rows causes little bias
MAR, Missing At Random	Missingness depends on other observed columns	Younger respondents skip the income question	Can be modelled using other columns
MNAR, Missing Not At Random	Missingness depends on the missing value itself	High earners refuse to report income	Hardest; simple fixes can bias results

💡 In MNAR cases, the fact that a value is missing is itself informative, so adding a "was missing" indicator can help.

6.2 Detecting Missing Values

df.isna().sum()                              # count per column
(df.isna().mean() * 100).round(1)            # percentage per column
df[df.isna().any(axis=1)]                    # rows with any missing value
df.isna().sum(axis=1).value_counts()         # how many missing per row

sns.heatmap(df.isna(), cbar=False)           # visual map of where the gaps are

Also hunt for disguised missing values: 0, -1, 999, "unknown", "N/A", "".

df = df.replace({"?": np.nan, "unknown": np.nan, -999: np.nan})

6.3 Strategy Overview

                    Missing values
                  ┌───────┴────────┐
              DELETE             IMPUTE (fill in)
         ┌──────┴──────┐      ┌─────────┼───────────────┐
      rows        columns   Simple   Group/Time-aware   Model-based
                          (mean,    (group mean,        (KNN,
                          median,   ffill, interpolate) iterative)
                          mode,
                          constant)

6.4 Deletion

df.dropna()                          # drop rows with ANY missing value
df.dropna(subset=["target"])         # drop rows missing in specific columns
df.dropna(thresh=5)                  # keep rows with at least 5 non-missing values
df.dropna(axis=1, thresh=int(0.6 * len(df)))      # drop columns that are >40% empty

✅ Delete when	❌ Avoid when
Very few rows affected (e.g., < 5%) and MCAR	Many rows are affected (you lose lots of data)
The target itself is missing	Missingness isn't random (introduces bias)
A column is almost entirely empty	The dataset is small

6.5 Simple Imputation

Strategy	Use for	Notes
Mean	Numeric, roughly symmetric	Sensitive to outliers
Median	Numeric, skewed or with outliers	A safer default
Mode (most frequent)	Categorical
Constant (e.g., "Unknown", 0)	When "missing" has a meaning	Makes missingness explicit

df["age"]    = df["age"].fillna(df["age"].median())
df["city"]   = df["city"].fillna(df["city"].mode()[0])
df["coupon"] = df["coupon"].fillna("None")

Drawbacks of simple imputation: it shrinks variance (many identical filled values) and can weaken correlations, and it ignores relationships with other features.

6.6 Smarter Imputation

Group-wise imputation

Use the typical value within a relevant group:

df["income"] = df["income"].fillna(df.groupby("job")["income"].transform("median"))

Time-series imputation

df["temp"] = df["temp"].ffill()                 # carry the last value forward
df["temp"] = df["temp"].bfill()                 # or pull the next value backward
df["temp"] = df["temp"].interpolate()           # draw a line between known points

KNN imputation: fill using the most similar rows

from sklearn.impute import KNNImputer
imputer = KNNImputer(n_neighbors=5)
X_filled = imputer.fit_transform(X)             # numeric data; scale first for fairness

Iterative (model-based) imputation

Predicts each column with missing values from the other columns, in rounds.

from sklearn.experimental import enable_iterative_imputer      # required (experimental feature)
from sklearn.impute import IterativeImputer
imputer = IterativeImputer(max_iter=10, random_state=42)
X_filled = imputer.fit_transform(X)

6.7 Missing-Value Indicators

Keep the information that a value was missing:

df["income_missing"] = df["income"].isna().astype(int)       # manual flag
df["income"] = df["income"].fillna(df["income"].median())

or let Scikit-learn do it:

from sklearn.impute import SimpleImputer
imputer = SimpleImputer(strategy="median", add_indicator=True)    # appends indicator columns

6.8 Scikit-learn's SimpleImputer

from sklearn.impute import SimpleImputer

num_imp = SimpleImputer(strategy="median")                    # "mean", "median", "most_frequent", "constant"
cat_imp = SimpleImputer(strategy="constant", fill_value="Unknown")

X_train_imp = num_imp.fit_transform(X_train)     # learn medians from TRAIN only
X_test_imp  = num_imp.transform(X_test)          # reuse the same medians on test

🚨 Leakage alert: compute fill values (mean, median, etc.) from the training set only, then apply to validation/test data. See Section 19.

6.9 Which Method Should I Use?

Situation	Suggestion
< 5% missing, MCAR	Deleting rows is acceptable
Numeric, skewed	Median
Categorical	Mode or a new "Unknown" category
Missingness likely informative	Add indicator column(s)
Time series	Forward fill / interpolation
Strong relationships between features	KNN or iterative imputation
> 60–70% missing in a column	Consider dropping the column
Target is missing	Drop those rows

ℹ️ Some tree-based models (such as Scikit-learn's HistGradientBoosting*) can handle NaN natively, so imputation isn't always mandatory. Most others need it.

---

7. Handling Duplicate Records

Duplicates are repeated records of the same real-world thing. They distort statistics, bias the model toward repeated cases, and (worst) can leak data between train and test sets.

7.1 Types of Duplicates

Type	Description	Example
Exact duplicates	Every column identical	Row inserted twice
Partial / key duplicates	Same key (ID, email), different other fields	Same customer with two addresses
Near (fuzzy) duplicates	Almost the same, small differences	"Jon Smith" vs "John Smith", extra spaces, different capitalisation

7.2 Why They Appear

Repeated data imports, system glitches, merging datasets, user double-submissions, web scraping the same page twice, or data entry variations.

7.3 Detecting Duplicates

df.duplicated().sum()                                   # number of exact duplicate rows
df[df.duplicated(keep=False)].sort_values("id")         # see ALL copies (keep=False marks every copy)

df.duplicated(subset=["email"]).sum()                   # duplicates by a key column
df["customer_id"].is_unique
df["customer_id"].value_counts().loc[lambda s: s > 1]   # which IDs repeat?

keep= options: "first" (default: marks all but the first), "last", False (marks all copies).

7.4 Removing Duplicates

df = df.drop_duplicates()                                       # exact duplicates
df = df.drop_duplicates(subset=["email"], keep="first")         # by key
df = df.drop_duplicates(subset=["email"], keep="last")          # keep the latest (if sorted by date)

Keep the most recent record:

df = df.sort_values("updated_at").drop_duplicates(subset="customer_id", keep="last")

7.5 Handling Near-Duplicates

First normalise the text so near-duplicates become exact duplicates:

df["name_clean"] = (
    df["name"].str.strip().str.lower().str.replace(r"\s+", " ", regex=True)
)
df["email_clean"] = df["email"].str.strip().str.lower()
df = df.drop_duplicates(subset=["email_clean"])

For stubborn cases, fuzzy matching (comparing string similarity) can flag likely duplicates for review. Always review matches rather than deleting blindly.

7.6 Merging Instead of Dropping

When duplicates hold complementary information, combine them:

df = df.groupby("customer_id").agg({
    "email": "first",
    "total_spent": "sum",
    "last_login": "max",
}).reset_index()

7.7 ⚠️ When Duplicates Are Legitimate

Not every repeat is an error:

• Two different customers who bought the same item at the same price
• Repeated measurements (the same patient measured on several days)
• Naturally repeated events (transactions)

Check with domain knowledge and an identifier or timestamp before removing.

7.8 Duplicates and Data Splitting

If a record appears in both training and test sets, your test score is artificially inflated: the model has effectively seen the answer.

✅ Remove duplicates before splitting, and for grouped data (e.g., several rows per patient), keep each group entirely on one side of the split (see Section 18).

---

8. Outlier Detection and Treatment

An outlier is a data point that differs markedly from the rest. It may be an error (typo, sensor fault) or a genuine rare event (fraud, a very rich customer).

8.1 Types of Outliers

Type	Meaning	Example
Point (global)	One value far from everything	Age = 250
Contextual	Unusual given the context	30 °C is normal in July but odd in January
Collective	A group of points is unusual together	A flat-lined sensor for 2 hours

Outliers can also be univariate (extreme in one variable) or multivariate (a normal-looking combination in each variable but unusual together, such as a 5-year-old with an income of ₹10 lakh).

8.2 Why They Matter

• Distort means, standard deviations and correlations
• Strongly affect sensitive models (e.g., linear regression, distance-based methods)
• Skew scaling (e.g., min-max scaling is dominated by extremes)
• But they can also be the most important points (fraud detection!)

8.3 Detection Methods

1) Visual

sns.boxplot(x=df["income"])                 # dots beyond the whiskers = candidates
sns.histplot(df["income"])                  # lone bars far from the mass
sns.scatterplot(data=df, x="age", y="income")      # find odd pairs (multivariate)

2) Z-score method (assumes roughly bell-shaped data)

z = (df["income"] - df["income"].mean()) / df["income"].std()
outliers = df[z.abs() > 3]                 # more than 3 standard deviations away

⚠️ The mean and std are themselves affected by outliers, so extreme values can hide themselves.

3) IQR method (robust, no normality assumption)

Q1 = df["income"].quantile(0.25)
Q3 = df["income"].quantile(0.75)
IQR = Q3 - Q1
lower = Q1 - 1.5 * IQR
upper = Q3 + 1.5 * IQR
outliers = df[(df["income"] < lower) | (df["income"] > upper)]

This is the rule behind box-plot whiskers. Use 3 × IQR instead of 1.5 to flag only extreme outliers.

4) Modified Z-score (uses the median, robust)

median = df["income"].median()
mad = (df["income"] - median).abs().median()            # Median Absolute Deviation
modified_z = 0.6745 * (df["income"] - median) / mad
outliers = df[modified_z.abs() > 3.5]

5) Percentile method

low, high = df["income"].quantile([0.01, 0.99])
outliers = df[(df["income"] < low) | (df["income"] > high)]

6) Model-based (good for multivariate outliers)

from sklearn.ensemble import IsolationForest
iso = IsolationForest(contamination=0.02, random_state=42)    # expected outlier fraction
labels = iso.fit_predict(df[["age", "income"]])               # -1 = outlier, 1 = normal
outliers = df[labels == -1]

Others: Local Outlier Factor (compares a point's local density to its neighbours'), DBSCAN clustering (points not in any cluster are noise), Elliptic Envelope (assumes Gaussian data).

Method	Best for	Weakness
Z-score	Bell-shaped, single feature	Distorted by the outliers themselves
IQR	Skewed data, single feature	Univariate only
Modified Z	Robust single-feature check	Univariate only
Isolation Forest / LOF	Many features, complex data	Needs tuning; harder to explain

8.4 Worked Example (IQR)

Data: [12, 14, 15, 16, 18, 19, 20, 95]

• Q1 ≈ 14.75, Q3 ≈ 19.25, IQR = 4.5
• Lower fence = 14.75 − 6.75 = 8.0; upper fence = 19.25 + 6.75 = 26.0
• 95 > 26 → flagged as an outlier ✅

8.5 Treatment Options

Option	How	When to use
Remove	Drop the rows	Clear errors, and you can spare the data
Cap / Winsorize	Clip to lower/upper limits	Keep the row but tame the extreme
Transform	Log, square-root, etc.	Right-skewed data with a long tail
Impute	Treat as missing, then fill	Impossible values (age = 250)
Separate modelling	Treat the outlier group differently	A distinct segment (e.g., VIP customers)
Keep	Do nothing	They're genuine and informative (fraud!)
Robust methods	Use outlier-tolerant models or scalers	Can't or shouldn't remove them

# Winsorizing / capping
df["income_capped"] = df["income"].clip(lower, upper)

# Log transform to tame a long right tail
df["income_log"] = np.log1p(df["income"])

# Treat impossible values as missing
df.loc[(df["age"] < 0) | (df["age"] > 120), "age"] = np.nan

8.6 Decision Guide

Is the value impossible or clearly an error?
 ├─ Yes → fix it, set to missing, or remove
 └─ No  → Is it a real but extreme value?
          ├─ Does your model suffer from it? ─ Yes → cap / transform / robust method
          └─ Is it the thing you want to detect (fraud, faults)? ─ Yes → KEEP it!

8.7 Cautions

• ❌ Don't delete outliers automatically. Investigate first.
• ❌ Learn limits (IQR fences, percentiles) from the training set only, then apply them to the test set
• ✅ Check outliers before scaling
• ✅ Document what you did and why

---

9. Categorical Data Encoding (Overview)

Most ML algorithms need numbers. Encoding converts categories (text labels) into numeric form without inventing false meaning.

⚠️ The core danger: Encoding Red=1, Blue=2, Green=3 tells a model that Green is "three times" Red and that Blue is "between" them. For unordered categories that is false information.

9.1 Choosing an Encoder at a Glance

Encoder	Output	Best for	Watch out
Label	One integer per class	The target variable; tree models on features (with care)	Implies order
Ordinal	Integers following a real order	Ordered categories (low/med/high)	Order must be specified
One-hot	One 0/1 column per category	Nominal, low cardinality	Many columns for high cardinality
Target (mean)	Number from target statistics	High-cardinality nominal	Leakage risk; needs cross-fitting and smoothing
Frequency / count	How often each category appears	High-cardinality, when frequency is informative	Different categories with equal counts collide
Binary / hashing	Compact binary codes or hashed buckets	Very high cardinality	Less interpretable

9.2 Cardinality Matters

Cardinality	Examples	Usual approach
Low (≈ < 10–15)	Gender, payment type	One-hot
Medium	Country, product category	One-hot (with rare-category grouping) or target encoding
High (hundreds+)	Zip code, user ID, product ID	Target/frequency/hashing; or group rare values into "Other"

Group rare categories:

counts = df["city"].value_counts()
rare = counts[counts < 20].index
df["city"] = df["city"].replace(rare, "Other")

9.3 Clean First!

df["city"] = df["city"].str.strip().str.lower()           # merge "Delhi", "delhi ", "DELHI"

Encode only after inconsistent spellings are fixed.

9.4 Unseen Categories

New data at prediction time may contain a category never seen in training. Plan for it (handle_unknown="ignore" for one-hot; an "unknown" code for ordinal).

---

10. Label Encoding and One-Hot Encoding

10.1 Label Encoding

Assigns each category a unique integer (0, 1, 2, …).

Cat → 0     Dog → 1     Bird → 2

from sklearn.preprocessing import LabelEncoder

le = LabelEncoder()
y_encoded = le.fit_transform(["cat", "dog", "bird", "dog"])     # [1 2 0 2]  (alphabetical: bird=0, cat=1, dog=2)
le.classes_                                                     # ['bird' 'cat' 'dog']
le.inverse_transform([1, 2])                                    # ['cat' 'dog']

Important notes

• LabelEncoder is designed for the target (y), one column at a time.
• For input features, use OrdinalEncoder (works on several columns, fits into pipelines).
• The integers create an artificial order. Linear and distance-based models will misread it. Tree-based models are less affected.

✅ Good for	❌ Bad for
Encoding class labels (target)	Nominal features in linear/distance-based models
Tree-based models (with care)	Anything where artificial order would mislead

10.2 One-Hot Encoding

Creates one new binary column per category: 1 if the row has that category, else 0.

 Colour     →   Red  Blue  Green
 Red        →    1    0     0
 Blue       →    0    1     0
 Green      →    0    0     1
 Blue       →    0    1     0

No false order is introduced: every category is equally "far" from the others.

With pandas

pd.get_dummies(df, columns=["colour"])
pd.get_dummies(df, columns=["colour"], drop_first=True, dtype=int)     # drop one column; 0/1 ints

With Scikit-learn (preferred for ML pipelines)

from sklearn.preprocessing import OneHotEncoder

ohe = OneHotEncoder(handle_unknown="ignore", sparse_output=False)
X_train_ohe = ohe.fit_transform(X_train[["colour"]])        # learn categories from TRAIN
X_test_ohe  = ohe.transform(X_test[["colour"]])             # same columns guaranteed on test

ohe.get_feature_names_out()                                 # ['colour_Blue' 'colour_Green' 'colour_Red']

Why Scikit-learn's encoder beats get_dummies for modelling: it remembers the categories from training, so the test data gets exactly the same columns, and unseen categories are handled gracefully (all zeros with handle_unknown="ignore"). With get_dummies, train and test could end up with different columns.

The Dummy Variable Trap

The columns are perfectly redundant: if Red = 0 and Blue = 0, then Green must be 1. This perfect multicollinearity can break linear models with no regularisation.

Fix: drop one column: drop="first" (or drop_first=True). With k categories, keep k − 1 columns. (Tree models and regularised models are not troubled by it.)

OneHotEncoder(drop="first", handle_unknown="ignore")      # note: combining drop with ignore has caveats in some versions

Pros and cons

✅ Pros	❌ Cons
No fake ordering	Many columns for high-cardinality features ("curse of dimensionality")
Simple and widely supported	Sparse, memory-heavy data
Works well for linear and distance-based models	Doesn't capture any relationship between categories

Tips

• Use for nominal features with low/medium cardinality
• Group rare categories first
• Keep it inside a pipeline (see Section 20)

10.3 Label vs One-Hot Compared

	Label Encoding	One-Hot Encoding
Output	1 column of integers	k binary columns
Implied order	✅ Yes (artificial)	❌ None
Dimensionality	Unchanged	Increases
Use for	Target; some tree models	Nominal features
Linear/distance models	⚠️ Misleading	✅ Suitable

---

11. Ordinal Encoding and Target Encoding

11.1 Ordinal Encoding

Used when categories have a natural, meaningful order. You assign integers that respect that order.

low → 0     medium → 1     high → 2

from sklearn.preprocessing import OrdinalEncoder

enc = OrdinalEncoder(categories=[["low", "medium", "high"]])       # you specify the order!
X_enc = enc.fit_transform(df[["priority"]])                        # [[0.] [1.] [2.]]

Handling several columns, each with its own order:

enc = OrdinalEncoder(
    categories=[["low", "medium", "high"], ["primary", "secondary", "graduate", "postgraduate"]],
    handle_unknown="use_encoded_value", unknown_value=-1,
)

Or with pandas:

order = {"low": 0, "medium": 1, "high": 2}
df["priority_num"] = df["priority"].map(order)

Key points

• ⚠️ Always specify the order. The default is alphabetical, which is usually wrong ("high" < "low" < "medium")
• The numeric gaps are an assumption: is "high" really twice "medium"? The encoding implies equal spacing
• Great for: education level, ratings, size (S/M/L/XL), severity

✅ Pros	❌ Cons
Compact: one column	Assumes equal spacing between levels
Preserves order information	Only valid for genuinely ordered categories

11.2 Target Encoding (Mean Encoding)

Replaces each category with a statistic of the target for that category, usually the mean target value.

Example (predicting whether a customer buys, 1 = bought):

City	Rows	Average bought	→ Encoded value
Pune	40	0.30	0.30
Delhi	100	0.55	0.55
Bhopal	20	0.40	0.40

means = df.groupby("city")["bought"].mean()
df["city_te"] = df["city"].map(means)

This packs a lot of information into one column, even for a category with hundreds of values, which is why it's popular for high-cardinality features.

⚠️ Danger 1: Target Leakage

The encoded value is computed using the target. If you compute means on the same rows you then train on, each row's own label influences its feature, so the model can "cheat", leading to overfitting.

Prevention:

• Compute the encoding on the training set only, then apply to validation/test
• Use cross-fitting (out-of-fold encoding): encode each fold using statistics computed from the other folds

⚠️ Danger 2: Rare Categories → Noisy Estimates

A category with 2 rows and a mean of 1.0 is unreliable. Smoothing blends the category mean with the overall mean:

encoded = ( n · category_mean  +  m · global_mean ) / ( n + m )

• n = number of rows in the category
• m = smoothing strength (higher → trust the global mean more)

Worked example: global mean = 0.40; a city with n = 10, category mean = 0.80; m = 10:

encoded = (10 × 0.80 + 10 × 0.40) / (10 + 10) = 12 / 20 = 0.60

The raw 0.80 is pulled toward 0.40 because only 10 rows back it up. With n = 1000, the estimate would stay near 0.80.

Using Scikit-learn's TargetEncoder (version 1.3+)

from sklearn.preprocessing import TargetEncoder

te = TargetEncoder(smooth="auto", cv=5, random_state=42)
X_train_te = te.fit_transform(X_train[["city"]], y_train)     # uses internal cross-fitting
X_test_te  = te.transform(X_test[["city"]])                   # uses encodings learned on all training data

fit_transform uses cross-fitting internally to reduce leakage. Because of that, fit(X, y).transform(X) isn't the same as fit_transform(X, y). Always use fit_transform on the training data.

Pros and cons

✅ Pros	❌ Cons
One column, even for high cardinality	Leakage and overfitting if done carelessly
Captures the category–target relationship directly	Needs smoothing for rare categories
Often strong for tree/boosting models	Requires the target, so it's supervised preprocessing

11.3 Other Handy Encodings

Encoding	Idea	Example
Frequency / count encoding	Replace with how often the category occurs	df["city"].map(df["city"].value_counts(normalize=True))
Binary encoding	Convert the integer code to binary digits across a few columns	100 categories → ~7 columns instead of 100
Hashing	Hash categories into a fixed number of buckets	Fixed width, handles unseen categories; collisions possible

11.4 Which Encoder? Summary

Is it the TARGET?               → LabelEncoder
Is there a natural ORDER?       → OrdinalEncoder (specify the order)
Unordered, few categories?      → OneHotEncoder
Unordered, MANY categories?     → Target / frequency / hashing (carefully!) or group rare values

---

12. Feature Scaling

Feature scaling puts numeric features on comparable ranges.

12.1 The Problem

Feature	Typical range
Age	18 – 70
Salary	20,000 – 2,000,000
Number of children	0 – 5

Without scaling, salary dominates simply because its numbers are bigger, not because it's more important.

12.2 Which Algorithms Need Scaling?

Need scaling ✅	Why
Distance-based (k-NN, k-Means, SVM with RBF kernel)	Distances are dominated by large-range features
Gradient-based (linear/logistic regression, neural networks)	Converges faster and more stably when features are on similar scales
Regularised models (Ridge, Lasso)	Penalties treat all coefficients equally, so unscaled features are penalised unfairly
PCA	Components are driven by variance, which depends on scale

Usually don't need scaling ❌	Why
Decision trees, Random Forests, Gradient Boosting	Splits depend on feature order, not magnitude
Naive Bayes (generally)	Works with per-feature distributions

(It doesn't hurt trees, it's just usually unnecessary.)

12.3 Visual Intuition

Before scaling (elongated)        After scaling (rounder)
  salary                            salary
    │      · ·                        │   · ·
    │    · ·                          │  · · ·
    │  · ·       Gradient descent     │  · · ·     Gradient descent
    │· ·         zig-zags slowly      │            heads straight to the minimum
    └──────── age                     └──────── age

12.4 Golden Rules

1. Fit the scaler on the training data only, then apply (transform) to validation and test data
2. Scale features, not the target (usually, and only if you have a specific reason)
3. Don't scale one-hot columns or other 0/1 indicators
4. Handle outliers first, since some scalers are very sensitive to them

---

13. Normalization and Standardization

🗣️ Terminology warning: "Normalization" is used loosely. It often means min-max scaling (rescale into a range), but in Scikit-learn the Normalizer class scales each row to unit length. In these notes: Normalization = rescaling to a range, Standardization = mean 0, std 1.

13.1 Min-Max Normalization

Rescales values to a fixed range, usually [0, 1].

x_scaled = (x − x_min) / (x_max − x_min)

from sklearn.preprocessing import MinMaxScaler

scaler = MinMaxScaler(feature_range=(0, 1))
X_train_s = scaler.fit_transform(X_train)
X_test_s  = scaler.transform(X_test)

✅ Pros	❌ Cons
Bounded, intuitive range	Very sensitive to outliers (one extreme squashes everyone else)
Preserves the shape of the distribution	Test data can fall outside [0, 1]
Good for bounded-input models (e.g., image pixels, some neural nets)

13.2 Standardization (Z-score Scaling)

Shifts to mean 0 and scales to standard deviation 1.

z = (x − μ) / σ

from sklearn.preprocessing import StandardScaler

scaler = StandardScaler()
X_train_s = scaler.fit_transform(X_train)
X_test_s  = scaler.transform(X_test)
scaler.mean_, scaler.scale_            # learned mean and standard deviation per feature

✅ Pros	❌ Cons
Not confined to a range, so less distorted by outliers than min-max	Doesn't produce a bounded range
Standard choice for many algorithms (linear models, SVM, PCA, neural nets)	Still affected by extreme outliers (mean and std shift)
Values mean "how many standard deviations from average"

Note: Scikit-learn's StandardScaler divides by the population standard deviation (ddof = 0).

13.3 Worked Comparison

Data: [10, 20, 30, 40, 100] (the 100 is an outlier)

Value	Min-Max	Standardized	Robust
10	0.000	−0.949	−1.0
20	0.111	−0.632	−0.5
30	0.222	−0.316	0.0
40	0.333	0.000	0.5
100	1.000	1.897	3.5

• Min-max: the outlier squeezes the four normal values into 0–0.33
• Standardized: mean = 40, std ≈ 31.6; the outlier still pulls the mean and std
• Robust: median = 30, IQR = 20, so the four normal values stay nicely spread out, and the outlier stays visibly extreme

13.4 Other Scalers

Scaler	Formula / idea	Use when
RobustScaler	(x − median) / IQR	Data has outliers
MaxAbsScaler	x / max(|x|) → range [−1, 1]	Sparse data (keeps zeros as zeros)
Normalizer	Scales each row (sample) to unit length (L2 norm = 1)	When only the direction matters (e.g., text vectors, cosine similarity)
Log / power scaling	See Section 15	Skewed data

from sklearn.preprocessing import RobustScaler, MaxAbsScaler, Normalizer
RobustScaler().fit_transform(X_train)
Normalizer(norm="l2").fit_transform(X)             # operates ROW by ROW (no fitting needed)

13.5 Normalization vs Standardization

	Min-Max Normalization	Standardization
Result range	[0, 1] (or chosen)	Unbounded; mean 0, std 1
Outlier sensitivity	High	Moderate
Distribution shape	Preserved	Preserved
Typical use	Images, bounded inputs, some neural nets	Linear/logistic regression, SVM, PCA, most general use
When unsure		✅ Good default

13.6 Quick Decision Guide

Outliers present?                       → RobustScaler (or treat outliers, then standardize)
Need values strictly in [0, 1]?         → MinMaxScaler
Sparse data (lots of zeros)?            → MaxAbsScaler
Roughly bell-shaped / general purpose?  → StandardScaler
Tree-based model?                       → Scaling usually unnecessary

13.7 Reminder: No Leakage

# ❌ WRONG: statistics from ALL data, including the test set
scaler.fit(X_all);  X_all_s = scaler.transform(X_all);  split afterwards

# ✅ RIGHT: split first, learn from the training set only
X_train, X_test = train_test_split(X, ...)
scaler.fit(X_train)
X_train_s = scaler.transform(X_train)
X_test_s  = scaler.transform(X_test)

---

14. Discretization and Binning

Discretization (binning) converts a continuous numeric feature into discrete intervals ("bins").

Age: 5, 17, 23, 34, 41, 58, 72
        ↓ bins: [0–18), [18–40), [40–65), [65+)
Group:  Child, Child, Adult, Adult, Middle, Middle, Senior

14.1 Why Bin?

• 🧩 Capture non-linear effects with simple models (e.g., "teenagers behave differently")
• 🔇 Reduce the influence of noise and small fluctuations
• 🎯 Tame outliers (extremes fall into the top bin)
• 📖 Make results interpretable ("age band 25–35")
• 🛠️ Some algorithms (e.g., Naive Bayes variants) work naturally with discrete features

The cost: information loss. Everything within a bin is treated as identical, so only bin when there's a reason.

14.2 Binning Strategies

Strategy	How bins are formed	Pros	Cons
Equal-width (uniform)	Same-sized intervals across the range	Simple, easy to explain	Skewed data → some bins nearly empty, one overloaded
Equal-frequency (quantile)	Each bin has about the same number of rows	Balanced bins; robust to outliers	Bin widths vary; similar values may be split
Custom / domain-based	You choose meaningful cut points	Most interpretable	Needs domain knowledge
Clustering-based (k-means)	Bins follow natural groupings	Adapts to structure	Less transparent

14.3 With pandas

ages = pd.Series([5, 17, 23, 34, 41, 58, 72])

# Equal-width: 3 bins of equal size
pd.cut(ages, bins=3)

# Custom bins with labels
pd.cut(ages, bins=[0, 18, 40, 65, 120],
       labels=["child", "young adult", "middle-aged", "senior"],
       right=False)                      # right=False → intervals like [0, 18)

# Equal-frequency: 4 quartile bins
pd.qcut(df["income"], q=4, labels=["Q1", "Q2", "Q3", "Q4"])

Function	Bins by	Result
pd.cut	Value ranges	Equal-width (or your own edges)
pd.qcut	Quantiles	Equal-frequency

14.4 With Scikit-learn

from sklearn.preprocessing import KBinsDiscretizer

kbd = KBinsDiscretizer(n_bins=4, encode="ordinal", strategy="quantile")
# strategy: "uniform" (equal width), "quantile" (equal frequency), "kmeans"
# encode:   "ordinal" (bin number), "onehot", "onehot-dense"

X_train_b = kbd.fit_transform(X_train[["age", "income"]])     # bin edges learned from TRAIN
X_test_b  = kbd.transform(X_test[["age", "income"]])
kbd.bin_edges_

Being a transformer, it slots into pipelines and avoids leakage.

14.5 Binning Then Encoding

Bins are ordinal categories. You can keep them as integers (bin 0, 1, 2…) or one-hot encode them (encode="onehot") to let a linear model learn a separate effect per bin, a way to capture non-linear shapes.

14.6 Choosing the Number of Bins

• Too few: lose detail (underfitting the feature)
• Too many: noisy, sparse bins (little benefit over the raw number)
• Start with a handful (e.g., 4–10), check the result visually (histogram of the binned feature), and tune

14.7 When Not to Bin

• Tree-based models already find their own split points on raw numbers, so binning rarely helps
• When the precise numeric value matters
• When you can't justify the information loss

---

15. Data Transformation

Data transformation changes the shape or representation of features (not just the scale) to make patterns easier for a model to learn, or to satisfy modelling assumptions.

15.1 Why Transform?

• Reduce skewness: bring long-tailed data closer to a symmetric (bell-like) shape
• Stabilise variance: so spread doesn't grow with the value
• Linearise relationships: so a linear model fits better
• Reduce the impact of outliers
• Satisfy assumptions of certain models (e.g., roughly normal residuals in linear regression)

15.2 Mathematical Transforms for Skewed Data

Right-skewed data (long tail →)         After log transform
 count                                   count
  █                                        ▃▆█▆▃
  █▅                                      
  █▇▃▂▁▁▁▁                                 (more symmetric)
  └──────────► income                      └──────────► log(income)

Transform	Formula	Use for	Notes
Log	log(x) or log1p(x) = log(1+x)	Strong right skew (income, prices, counts)	log needs x > 0; log1p handles 0
Square root	√x	Moderate right skew, counts	Needs x ≥ 0
Reciprocal	1/x	Very strong skew, rates	Reverses order
Square / cube	x², x³	Left skew
Box-Cox	(xλ − 1)/λ (finds best λ)	Positive data; automatically picks the power	Requires x > 0
Yeo-Johnson	Box-Cox extension	Data with zero or negative values	More flexible

df["income_log"]  = np.log1p(df["income"])
df["income_sqrt"] = np.sqrt(df["income"])

# Check improvement
df["income"].skew(), df["income_log"].skew()

15.3 Power and Quantile Transformers (Scikit-learn)

from sklearn.preprocessing import PowerTransformer, QuantileTransformer

pt = PowerTransformer(method="yeo-johnson")          # or "box-cox" (positive data only)
X_train_t = pt.fit_transform(X_train)                # learns the best λ per feature from TRAIN
X_test_t  = pt.transform(X_test)                     # also standardizes the output by default

qt = QuantileTransformer(output_distribution="normal", n_quantiles=200, random_state=42)
X_train_q = qt.fit_transform(X_train)                # forces a normal (or uniform) shape

Transformer	What it does	Note
PowerTransformer	Finds a power transform making data more Gaussian	Smooth and invertible
QuantileTransformer	Maps values by rank onto a chosen distribution	Strongly reduces outlier impact; can distort distances between values

15.4 Transforming the Target

For a skewed numeric target (e.g., house prices), you can model log(price) and convert back afterwards:

from sklearn.compose import TransformedTargetRegressor
from sklearn.linear_model import LinearRegression

model = TransformedTargetRegressor(
    regressor=LinearRegression(), func=np.log1p, inverse_func=np.expm1
)
model.fit(X_train, y_train)             # trains on log1p(y); predictions come back in original units

15.5 Creating New Features from Existing Ones

Polynomial and interaction features

Let linear models learn curves and combined effects:

from sklearn.preprocessing import PolynomialFeatures

poly = PolynomialFeatures(degree=2, include_bias=False)
poly.fit_transform([[2, 3]])           # [[2, 3, 4, 6, 9]]  → x1, x2, x1², x1·x2, x2²

(Feature count grows quickly with degree. Use sparingly and scale afterwards.)

Date/time decomposition

df["year"]    = df["date"].dt.year
df["month"]   = df["date"].dt.month
df["weekday"] = df["date"].dt.dayofweek
df["is_weekend"] = (df["weekday"] >= 5).astype(int)

Cyclical features

Months, hours and weekdays wrap around (December is next to January). Encoding month as 1–12 puts 12 and 1 far apart. Use sine and cosine:

df["month_sin"] = np.sin(2 * np.pi * df["month"] / 12)
df["month_cos"] = np.cos(2 * np.pi * df["month"] / 12)

Ratios, differences, aggregates

df["price_per_sqft"] = df["price"] / df["area"]
df["age_at_purchase"] = df["purchase_year"] - df["birth_year"]

Custom transforms in a pipeline

from sklearn.preprocessing import FunctionTransformer
log_tf = FunctionTransformer(np.log1p, inverse_func=np.expm1)

15.6 Transformation vs Scaling

	Scaling (Section 12–13)	Transformation (this section)
Changes	Range / units	Shape of the distribution (and sometimes feature meaning)
Distribution shape	Preserved	Altered
Example	Standardize income	Log-transform income

They're often combined: transform to reduce skew → then scale.

15.7 Cautions

• Check why you're transforming. Look at histograms before and after
• Remember to apply the same transformation (with parameters learned on training data) to test data
• Transformed features are harder to interpret. Keep track of what you did
• Avoid log of zero or negative numbers (use log1p or Yeo-Johnson)
• Tree-based models are generally insensitive to monotonic transforms of features

---

16. Handling Imbalanced Datasets

A dataset is imbalanced when some classes appear far less often than others.

Domain	Typical minority class	Typical share
Fraud detection	Fraudulent transactions	< 1%
Medical screening	Disease present	A few %
Equipment monitoring	Failure	Rare
Spam / churn	Spam / churners	Often 5–20%

16.1 Why It's a Problem

Models learn to favour the majority class, because predicting it is "usually right."

The accuracy trap: With 99% legitimate and 1% fraudulent transactions, a model that always says "legitimate" gets 99% accuracy, yet catches zero fraud. It's useless.

16.2 Step 1: Use the Right Metrics

Metric	Why it helps
Precision	Of the cases flagged positive, how many truly are?
Recall (sensitivity)	Of all true positives, how many did we catch?
F1-score	Balance of precision and recall
PR-AUC (Precision–Recall AUC)	Summarises the precision/recall trade-off, informative under heavy imbalance
ROC-AUC	Ranking quality across thresholds (can look optimistic when imbalance is extreme)
Confusion matrix	Shows exactly where mistakes happen
Balanced accuracy	Average of recall per class

from sklearn.metrics import classification_report, confusion_matrix, average_precision_score, balanced_accuracy_score
print(classification_report(y_test, y_pred))

Which to prioritise depends on the cost of errors:

• Missing a real case is costly (cancer, fraud) → favour recall
• False alarms are costly (blocking legitimate customers) → favour precision

16.3 Step 2: Split Carefully

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=42       # keeps the class ratio in both sets
)

Use stratified splits and stratified cross-validation (StratifiedKFold) so each fold contains minority examples.

16.4 Step 3: Techniques to Balance Learning

A) Data-level: Resampling

Original:        ■■■■■■■■■■■■■■■■■■ □□
Undersample:     ■■■■ □□                  (remove majority examples)
Oversample:      ■■■■■■■■■■■■■■■■■■ □□□□□□□□□□□□□□□□□□  (add minority examples)

Technique	Idea	Pros	Cons
Random undersampling	Remove random majority rows	Fast; smaller data	Throws away information
Random oversampling	Duplicate minority rows	No information loss	Can cause overfitting (exact copies)
SMOTE	Create synthetic minority samples by interpolating between neighbouring minority points	Less overfitting than duplication	Can create noisy or overlapping samples; numeric features only (variants exist for categorical)
Tomek links / ENN (cleaning)	Remove ambiguous majority points near the boundary	Cleaner class boundary	May remove useful data
Combined (e.g., SMOTE + cleaning)	Oversample, then clean	Often works well	More complex

How SMOTE works:

1. Pick a minority point.
2. Find its k nearest minority neighbours.
3. Choose one neighbour at random and create a new point somewhere on the line between them.

# pip install imbalanced-learn
from imblearn.over_sampling import SMOTE, RandomOverSampler
from imblearn.under_sampling import RandomUnderSampler

smote = SMOTE(random_state=42)
X_res, y_res = smote.fit_resample(X_train, y_train)       # resample the TRAINING set only!

🚨 Resample the training data ONLY, never the validation/test data. The test set should reflect the real-world class balance. Resampling before splitting leaks synthetic copies of test-related points into training.

B) Algorithm-level: Class weights

Make mistakes on the minority class cost more, with no change to the data:

from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier

LogisticRegression(class_weight="balanced")
RandomForestClassifier(class_weight="balanced", random_state=42)
LogisticRegression(class_weight={0: 1, 1: 10})        # custom: minority errors count 10×

"balanced" sets weights inversely proportional to class frequencies. It's often the simplest first thing to try.

C) Threshold moving

Models output probabilities, and the default cut-off is 0.5. For rare positives, lower the threshold to catch more of them:

proba = model.predict_proba(X_test)[:, 1]
y_pred = (proba >= 0.3).astype(int)

Choose the threshold using validation data and the precision–recall trade-off.

D) Other approaches

• Collect more minority data (often the best fix if possible)
• Ensemble methods designed for imbalance (e.g., balanced bagging/boosting variants)
• Anomaly detection framing when the minority class is extremely rare
• Data augmentation for the minority class (e.g., for images)

16.5 Resampling Inside a Pipeline (No Leakage)

from imblearn.pipeline import Pipeline as ImbPipeline
from sklearn.preprocessing import StandardScaler

pipe = ImbPipeline([
    ("scale", StandardScaler()),
    ("smote", SMOTE(random_state=42)),            # applied only when FITTING, i.e. on training folds
    ("model", LogisticRegression(max_iter=500)),
])
scores = cross_val_score(pipe, X, y, cv=StratifiedKFold(5), scoring="f1")

Using the imblearn Pipeline ensures SMOTE is applied only to the training portion of each fold.

16.6 Choosing an Approach

Situation	Try first
Mild imbalance	Stratified split + class weights + proper metrics
Moderate imbalance, enough data	Class weights, or random undersampling
Severe imbalance, small minority	SMOTE (in a pipeline) / class weights / threshold tuning
Costs of errors differ a lot	Threshold tuning + cost-sensitive metrics
Possible with more data	Collect more minority examples

---

17. Sampling Techniques

Sampling means selecting a subset of data from a larger set. We sample because the full dataset may be too large to process, too expensive to collect or label, or just unnecessary.

17.1 Population vs Sample

• Population: the complete group you care about
• Sample: the subset you actually work with
• Goal: a sample that is representative (a "miniature" of the population)
• Sampling frame: the list from which you draw the sample
• Sampling error: natural difference between sample and population (shrinks as sample size grows)
• Sampling bias: systematic distortion (doesn't shrink with size)

Population ──(sampling method)──► Sample ──(analysis/model)──► Conclusions about the population

17.2 Probability Sampling (Random, Every Member Has a Known Chance)

Method	How it works	Example	Pros	Cons
Simple random	Every item equally likely	Draw 1,000 customers at random	Unbiased, simple	May under-represent small groups
Systematic	Take every k-th item after a random start	Every 10th record	Easy, evenly spread	Problem if the data has a periodic pattern matching k
Stratified	Split into groups (strata), sample from each	Sample proportionally from each region or class	Guarantees representation of every group; more precise	Needs known group labels
Cluster	Randomly pick whole groups, use all members	Choose 5 schools, survey all students	Cheaper when groups are natural	Higher error if clusters differ from each other
Multistage	Sample in stages (clusters, then individuals within)	Cities → neighbourhoods → households	Practical for huge populations	More complex

# Simple random sample
df.sample(n=1000, random_state=42)                  # exactly 1,000 rows
df.sample(frac=0.1, random_state=42)                # 10% of rows

# Systematic sample
df.iloc[::10]                                       # every 10th row

# Stratified sample (keep proportions of a column)
df.groupby("region", group_keys=False).apply(
    lambda g: g.sample(frac=0.1, random_state=42)
)
# or while splitting data:
train_test_split(X, y, test_size=0.2, stratify=y, random_state=42)

# Cluster sample
chosen = np.random.default_rng(42).choice(df["school"].unique(), size=5, replace=False)
sample = df[df["school"].isin(chosen)]

17.3 Non-Probability Sampling (Not Random, So Bias Is a Risk)

Method	How it works	Typical use	Risk
Convenience	Whoever is easiest to reach	Quick pilot surveys	Highly biased
Quota	Fill fixed quotas per group, non-randomly	Market research	Selection within the quota is non-random
Purposive (judgement)	Researcher picks specific cases	Expert panels, rare cases	Subjective
Snowball	Participants recruit others	Hard-to-reach groups	Network bias

Fine for exploration, but be cautious about generalising from them.

17.4 Resampling Techniques

Different from the above: here you re-draw from the data you already have.

Bootstrap

Sample with replacement to create many "new" datasets of the same size; useful for estimating the uncertainty of a statistic or model, and the basis of bagging.

rng = np.random.default_rng(42)
boot_means = [rng.choice(data, size=len(data), replace=True).mean() for _ in range(1000)]
np.percentile(boot_means, [2.5, 97.5])               # rough 95% interval for the mean

With replacement, each bootstrap sample leaves out roughly 36.8% of the original rows (the out-of-bag samples), which can be used for validation.

With vs without replacement

	Without replacement	With replacement
Same item picked twice?	No	Yes
Typical use	Standard samples, splits	Bootstrap, oversampling

Class-balancing resampling

Over/under-sampling (see Section 16).

Cross-validation folds

A systematic form of resampling used to evaluate models (see Section 18).

17.5 Sampling Special Cases

• Time-ordered data: don't randomly shuffle if the future must not influence the past. Sample or split by time
• Streaming data: reservoir sampling keeps a fixed-size random sample while data keeps arriving
• Grouped data: sample by group (e.g., patients), not by row, to keep each group intact

17.6 How Big Should the Sample Be?

There's no single number. It depends on:

• The variability of the data (more variation → more samples)
• The number of features and model complexity
• The rarity of the classes you care about (you need enough minority examples)
• Desired precision (smaller margin of error requires more data)

Rule of thumb: more data helps, but representativeness matters more than raw size. A huge biased sample is still biased. Plot a learning curve (performance vs training size) to see whether more data would help.

17.7 Sampling Summary

Need every group represented?          → Stratified
Data naturally in clusters?            → Cluster
Quick and simple, no group structure?  → Simple random
Regular list, no hidden pattern?       → Systematic
Estimating uncertainty?                → Bootstrap
Class imbalance?                       → Over / under-sampling (training set only)
Time series?                           → Sample/split chronologically

---

18. Train, Validation and Test Splits

A model that is judged on the same data it learned from will look brilliant, because it can memorise. To measure real-world performance, we hold data back.

📚 Analogy: The training set is the textbook and practice problems. The validation set is a mock exam you use to adjust your study strategy. The test set is the final exam you take once, with questions you've never seen.

18.1 The Three Sets

Set	Purpose	Used for	Touched how often?
Training	The model learns from it	Fitting parameters	Constantly
Validation	Tune and compare models	Hyperparameters, model selection, early stopping	Many times
Test	Final, unbiased performance estimate	Reporting how it will do in the real world	Once, at the very end

 ┌───────────────────────────── Full dataset ─────────────────────────────┐
 │        Training (≈60–80%)         │  Validation (≈10–20%) │ Test (≈10–20%) │
 └───────────────────────────────────────────────────────────────────────┘
        fit the model                 tune & choose            final report

🚨 The test set is sacred. If you keep peeking at it and adjusting based on the result, it effectively becomes a validation set, and your final estimate becomes over-optimistic.

18.2 Typical Ratios

Dataset size	Common split
Small (thousands)	70 / 15 / 15 or 80 / 20 with cross-validation
Medium	60 / 20 / 20
Very large (millions)	98 / 1 / 1 can be enough (1% is still a lot of rows)

Make sure each set is large enough to give a stable estimate, especially for the minority class.

18.3 A Simple Train/Test Split

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(
    X, y,
    test_size=0.2,            # 20% held out
    random_state=42,          # reproducible
    shuffle=True,             # default; set False for time-ordered data
    stratify=y,               # keep class proportions (classification)
)

18.4 Train / Validation / Test (Three-Way Split)

Split twice:

# Step 1: carve out the test set (20%)
X_temp, X_test, y_temp, y_test = train_test_split(
    X, y, test_size=0.20, stratify=y, random_state=42
)

# Step 2: split the remainder into train (60% of total) and validation (20% of total)
X_train, X_val, y_train, y_val = train_test_split(
    X_temp, y_temp, test_size=0.25, stratify=y_temp, random_state=42   # 0.25 × 80% = 20%
)
# Result: 60% train / 20% validation / 20% test

18.5 Choosing the Right Splitting Strategy

Data situation	Strategy	Tool
General tabular, independent rows	Random split	train_test_split
Classification, imbalanced	Stratified split	stratify=y
Several rows per entity (patient, user)	Group-based split (an entity is entirely in one set)	GroupShuffleSplit, GroupKFold
Time series / future prediction	Chronological split (train on past, test on future)	Sort by time, or TimeSeriesSplit

Group split

Prevents the same patient/customer appearing in both train and test:

from sklearn.model_selection import GroupShuffleSplit

gss = GroupShuffleSplit(n_splits=1, test_size=0.2, random_state=42)
train_idx, test_idx = next(gss.split(X, y, groups=df["patient_id"]))
X_train, X_test = X.iloc[train_idx], X.iloc[test_idx]
y_train, y_test = y.iloc[train_idx], y.iloc[test_idx]

Time-based split

df = df.sort_values("date")
cutoff = int(len(df) * 0.8)
train, test = df.iloc[:cutoff], df.iloc[cutoff:]          # past → train, future → test

Using a random split on time series lets the model "see the future" and gives unrealistically good scores.

18.6 Cross-Validation (CV)

When data is limited, a single validation split can be unreliable. K-fold cross-validation rotates the validation role among k parts:

Fold 1: [VAL ][train][train][train][train]
Fold 2: [train][VAL ][train][train][train]
Fold 3: [train][train][VAL ][train][train]
Fold 4: [train][train][train][VAL ][train]
Fold 5: [train][train][train][train][VAL ]
                      → average the 5 scores

from sklearn.model_selection import cross_val_score, KFold, StratifiedKFold

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
scores = cross_val_score(pipeline, X_train, y_train, cv=cv, scoring="accuracy")
print(scores.mean(), scores.std())

Variant	Use when
KFold	Standard regression/classification
StratifiedKFold	Classification, especially imbalanced
GroupKFold	Grouped data
TimeSeriesSplit	Time-ordered data (always trains on earlier data, validates on later)
Leave-One-Out	Tiny datasets (expensive)

Typical workflow with CV:

1. Hold out a test set first.
2. Use cross-validation on the training set for model selection and tuning (so you don't need a separate validation set).
3. Retrain the best model on the whole training set.
4. Evaluate once on the test set.

18.7 Reading the Results

Observation	Likely meaning
Train score ≈ validation score, both high	Good fit 🎉
Train high, validation much lower	Overfitting
Both low	Underfitting (or poor features/data)
Validation good, test much worse	Over-tuning to the validation set, or leakage

18.8 Rules of Thumb

• ✅ Split first, before computing any statistics or fitting any transformer
• ✅ Always set random_state for reproducibility
• ✅ Make the test set resemble real future data
• ✅ Check that the class balance is similar across the sets
• ❌ Don't tune on the test set
• ❌ Don't let duplicates or related rows straddle the split

---

19. Data Leakage and Prevention

Data leakage is when information from outside the training data, usually from the test set or the future, sneaks into model training. The model then looks great in testing but fails in the real world.

🎓 Analogy: A student who obtains the exam questions in advance scores 100%, but has learned nothing about the subject.

19.1 Why It's So Dangerous

• Results look too good to be true and often are
• It's silent: no error message
• The failure only appears after deployment, which is expensive

19.2 The Main Types

1) Train–Test Contamination (preprocessing leakage)

Using the whole dataset to compute statistics for scaling, imputing, encoding, or feature selection before splitting.

# ❌ LEAKY: scaler learns mean/std from ALL rows, including the future test rows
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)
X_train, X_test, y_train, y_test = train_test_split(X_scaled, y)

# ✅ CORRECT: learn from training data only
X_train, X_test, y_train, y_test = train_test_split(X, y)
scaler = StandardScaler().fit(X_train)
X_train_s = scaler.transform(X_train)
X_test_s  = scaler.transform(X_test)

Same applies to: imputers, encoders (especially target encoding), outlier limits, PCA, SMOTE, feature selection.

2) Target Leakage

A feature contains information that wouldn't be available at prediction time, or is a consequence of the target.

Task	Leaky feature	Why it's a problem
Predict loan default	collections_status	Only exists after someone defaults
Predict hospital readmission	discharge_notes_after_followup	Written after the outcome
Predict customer churn	account_closed_date	Is the target, restated
Predict fraud	chargeback_amount	Known only after fraud is confirmed
Predict a sale	refund_issued	Happens after the sale

Ask for every feature: "Will I truly have this value at the moment I make a prediction?"

3) Temporal Leakage (Using the Future)

• Randomly splitting time-ordered data so future rows land in training
• Features computed using future information (e.g., a "30-day average" that includes days after the prediction date)

4) Duplicate / Overlap Leakage

The same or near-identical record appears in both train and test. The model simply memorises it.

5) Group Leakage

Related rows (same patient, user, device, document) are split across train and test. The model learns the entity, not the general pattern.

6) Leakage Through Hyperparameter Tuning

Repeatedly adjusting the model based on test-set performance. The test set silently becomes part of training.

7) Leakage in Resampling/Augmentation

Oversampling (e.g., SMOTE) or augmenting data before splitting, so near-copies of the same record appear on both sides.

19.3 Warning Signs 🚩

• Performance is suspiciously high (e.g., 99% on a hard problem)
• One feature has an enormous importance or an almost perfect correlation with the target
• Big gap between test performance and real-world performance
• Performance collapses when a particular feature is removed
• A feature's values are only filled in for one class

19.4 Prevention Checklist ✅

1. Split first. Before fitting any preprocessing step
2. Fit on train only. fit / fit_transform on training data; transform on validation/test
3. Use pipelines, so every step is learned within the training portion of each split/fold (see Section 20)
4. Cross-validate the whole pipeline, not just the model
5. Examine every feature for availability at prediction time (consult domain experts)
6. Respect time: use chronological splits and "as-of" feature calculations for temporal data
7. Deduplicate before splitting; split by group when rows are related
8. Resample only the training set (and inside the pipeline)
9. Keep the test set locked until the final evaluation
10. Be suspicious of great results. Investigate before celebrating

19.5 Using a Pipeline to Prevent Preprocessing Leakage

from sklearn.pipeline import make_pipeline
from sklearn.model_selection import cross_val_score

pipe = make_pipeline(SimpleImputer(strategy="median"), StandardScaler(), LogisticRegression())

# During cross-validation, the imputer and scaler are re-fitted on each fold's TRAINING part only.
scores = cross_val_score(pipe, X, y, cv=5)

Compare with the leaky way (scaling everything first, then cross-validating), where each validation fold has "influenced" the scaler.

19.6 Leakage Detection Quick Test

# Which single feature predicts the target "too well"?
for col in X.select_dtypes("number"):
    corr = np.abs(np.corrcoef(X[col].fillna(0), y)[0, 1])
    if corr > 0.9:
        print("Suspicious:", col, round(corr, 3))

A near-perfect predictor deserves a "how could I possibly know this in advance?" question.

---

20. Data Pipelines Using Scikit-learn

A pipeline chains preprocessing steps and a model into one object. You call fit() once and predict() once, and everything happens in the right order, with no leakage.

20.1 Why Use Pipelines?

Benefit	Explanation
🔒 Prevents leakage	Steps are fitted on training data only (including inside cross-validation)
🔁 Reproducible	Same steps applied identically to new data
🧼 Cleaner code	One object instead of many loose variables
🎛️ Easy tuning	Tune preprocessing and model hyperparameters together
🚀 Easy deployment	Save one object; new raw data goes straight in

20.2 Building Blocks

Tool	Purpose
Pipeline / make_pipeline	Chain steps in sequence (transformers, then a final model)
ColumnTransformer	Apply different preprocessing to different columns
make_column_selector	Pick columns by dtype/name automatically
FunctionTransformer	Wrap any function as a transformer
Custom transformer	Your own class (BaseEstimator + TransformerMixin)
FeatureUnion	Run transformers in parallel and join their outputs

20.3 A Basic Pipeline

from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression

pipe = Pipeline(steps=[
    ("imputer", SimpleImputer(strategy="median")),
    ("scaler",  StandardScaler()),
    ("model",   LogisticRegression(max_iter=500)),
])

pipe.fit(X_train, y_train)          # imputer.fit_transform → scaler.fit_transform → model.fit
y_pred = pipe.predict(X_test)       # imputer.transform → scaler.transform → model.predict
pipe.score(X_test, y_test)

Shortcut: make_pipeline names the steps automatically:

from sklearn.pipeline import make_pipeline
pipe = make_pipeline(SimpleImputer(strategy="median"), StandardScaler(), LogisticRegression())

Rules: every step except the last must be a transformer (has fit / transform). The last can be a transformer or a model.

Accessing parts:

pipe.named_steps["scaler"].mean_
pipe[0]                  # first step
pipe[:-1]                # everything except the model (just the preprocessing)

20.4 ColumnTransformer: Different Treatment per Column

Real tables mix numeric and categorical columns. Each needs its own steps.

from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import OneHotEncoder, OrdinalEncoder

numeric_features     = ["age", "income"]
categorical_features = ["city", "device"]
ordinal_features     = ["education"]

numeric_pipe = Pipeline([
    ("impute", SimpleImputer(strategy="median")),
    ("scale",  StandardScaler()),
])

categorical_pipe = Pipeline([
    ("impute", SimpleImputer(strategy="most_frequent")),
    ("onehot", OneHotEncoder(handle_unknown="ignore")),
])

ordinal_pipe = Pipeline([
    ("impute", SimpleImputer(strategy="most_frequent")),
    ("ordinal", OrdinalEncoder(categories=[["school", "graduate", "postgraduate"]])),
])

preprocessor = ColumnTransformer(
    transformers=[
        ("num", numeric_pipe,     numeric_features),
        ("cat", categorical_pipe, categorical_features),
        ("ord", ordinal_pipe,     ordinal_features),
    ],
    remainder="drop",          # what to do with unlisted columns: "drop" (default) or "passthrough"
)

Selecting columns automatically

from sklearn.compose import make_column_selector as selector

preprocessor = ColumnTransformer([
    ("num", numeric_pipe,     selector(dtype_include="number")),
    ("cat", categorical_pipe, selector(dtype_exclude="number")),
])

20.5 A Complete End-to-End Pipeline

import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier

X = df.drop(columns=["target"])
y = df["target"]

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=42
)

full_pipeline = Pipeline([
    ("preprocess", preprocessor),
    ("model", RandomForestClassifier(class_weight="balanced", random_state=42)),
])

full_pipeline.fit(X_train, y_train)               # raw DataFrame in...
print(full_pipeline.score(X_test, y_test))        # ...predictions out
full_pipeline.predict(new_raw_dataframe)          # new raw data goes straight in

20.6 Cross-Validation and Tuning with Pipelines

from sklearn.model_selection import cross_val_score, GridSearchCV

# Leak-free cross-validation: preprocessing is re-fitted inside each fold
cross_val_score(full_pipeline, X_train, y_train, cv=5, scoring="f1")

# Tune preprocessing AND model together: parameter names use  stepname__param
param_grid = {
    "preprocess__num__impute__strategy": ["mean", "median"],
    "model__n_estimators": [100, 300],
    "model__max_depth": [None, 10],
}
search = GridSearchCV(full_pipeline, param_grid, cv=5, scoring="f1")
search.fit(X_train, y_train)
search.best_params_
search.best_estimator_.score(X_test, y_test)          # final one-time test evaluation

Double underscores __ dive into nested steps: preprocess → num → impute → strategy.

20.7 Custom Transformers

Make your own step by subclassing BaseEstimator and TransformerMixin:

from sklearn.base import BaseEstimator, TransformerMixin

class OutlierCapper(BaseEstimator, TransformerMixin):
    """Clip each feature to IQR-based limits learned from the training data."""

    def __init__(self, factor=1.5):
        self.factor = factor

    def fit(self, X, y=None):
        X = np.asarray(X, dtype=float)
        q1, q3 = np.percentile(X, [25, 75], axis=0)
        iqr = q3 - q1
        self.lower_ = q1 - self.factor * iqr          # learned values end with "_"
        self.upper_ = q3 + self.factor * iqr
        return self                                   # always return self

    def transform(self, X):
        return np.clip(np.asarray(X, dtype=float), self.lower_, self.upper_)

numeric_pipe = Pipeline([
    ("impute", SimpleImputer(strategy="median")),
    ("cap",    OutlierCapper(factor=1.5)),
    ("scale",  StandardScaler()),
])

Pattern: learn from training data in fit, apply in transform. That's exactly what makes it leak-free.

Quick function-based step

from sklearn.preprocessing import FunctionTransformer
log_step = FunctionTransformer(np.log1p, feature_names_out="one-to-one")

20.8 Pipelines Including Transformations from This Guide

numeric_skewed = ["income", "spend"]
numeric_normal = ["age"]

preprocessor = ColumnTransformer([
    ("skewed", Pipeline([
        ("impute", SimpleImputer(strategy="median")),
        ("power",  PowerTransformer(method="yeo-johnson")),      # reduce skew, then standardises
    ]), numeric_skewed),

    ("normal", Pipeline([
        ("impute", SimpleImputer(strategy="mean")),
        ("scale",  StandardScaler()),
    ]), numeric_normal),

    ("binned", KBinsDiscretizer(n_bins=5, encode="onehot", strategy="quantile"), ["tenure"]),

    ("cat", OneHotEncoder(handle_unknown="ignore"), ["city"]),
])

20.9 Handy Features

# 1. See the pipeline as a diagram (in notebooks)
from sklearn import set_config
set_config(display="diagram")
full_pipeline

# 2. Get output column names
preprocessor.get_feature_names_out()

# 3. Get a DataFrame out of transformers (Scikit-learn 1.2+)
preprocessor.set_output(transform="pandas")
X_train_df = preprocessor.fit_transform(X_train)

# 4. Inspect all parameters
full_pipeline.get_params()

# 5. Save and reload the whole thing
import joblib
joblib.dump(full_pipeline, "pipeline.joblib")
loaded = joblib.load("pipeline.joblib")
loaded.predict(new_data)

20.10 Pipelines with Imbalanced Data

Use imblearn.pipeline.Pipeline when the pipeline contains a resampler such as SMOTE (see Section 16.5). The Scikit-learn Pipeline doesn't support steps that change the number of rows.

20.11 Pipeline Pitfalls

Pitfall	Fix
Fitting preprocessing outside the pipeline	Put every learned step inside the pipeline
Wrong column names in ColumnTransformer	Check X.columns; use selectors
Unseen categories at prediction time	OneHotEncoder(handle_unknown="ignore")
Forgetting remainder="passthrough" and losing columns	Set remainder deliberately
Doing target encoding or resampling outside	Keep them inside (and fitted on training data only)
Pipelines hiding what happens	Inspect intermediate outputs with pipe[:-1].transform(X)

20.12 The Ideal Workflow

1. Load raw data
2. Do EDA and quality checks (on training data)
3. Split off the TEST set (stratified / grouped / time-based as appropriate)
4. Define preprocessing in a ColumnTransformer
5. Wrap preprocessing + model in a Pipeline
6. Cross-validate / tune the whole pipeline on the TRAINING set
7. Fit the best pipeline on all training data
8. Evaluate ONCE on the test set
9. Save the pipeline and deploy it

---

21. Quick Revision Cheat Sheet

🔷 Part A: Understanding Data

Topic	Remember
Data types	Numerical (discrete/continuous), Categorical (nominal/ordinal), Date, Text. IDs and zip codes are categories
Formats	CSV (universal), JSON (nested), Parquet (fast and typed), SQL (databases)
Collection	Primary vs secondary; databases, APIs, scraping, sensors; document it and check bias and legality
Structure	Structured = tables; Semi = JSON/XML (flatten with json_normalize); Unstructured = text/images (extract features)
EDA	Overview → univariate → bivariate → multivariate → target. Look for skew, outliers, missingness, correlations
Quality	Completeness, accuracy, consistency, uniqueness, validity, timeliness

🔷 Part B: Cleaning

Topic	Remember
Missing	MCAR / MAR / MNAR. Delete (few, random) or impute (median / mode / KNN / iterative). Add indicator columns. Fit imputers on train only
Duplicates	duplicated(), drop_duplicates(subset=...). Normalise text first. Dedupe before splitting
Outliers	Detect: IQR (1.5×), z-score (`

🔷 Part C: Transforming Features

Topic	Remember
Label	For the target (LabelEncoder)
One-hot	Nominal, low cardinality. Use OneHotEncoder(handle_unknown="ignore"). Avoid dummy-variable trap with drop="first" for linear models
Ordinal	Ordered categories. Specify the order
Target encoding	High cardinality. Use smoothing + cross-fitting to avoid leakage
Scaling	Needed by distance/gradient/regularised models and PCA; not by trees
Min-Max	(x−min)/(max−min) → [0,1]; outlier-sensitive
Standardize	(x−μ)/σ → mean 0, std 1; good default
Robust	(x−median)/IQR; for outliers
Binning	pd.cut (equal-width), pd.qcut (equal-frequency), KBinsDiscretizer; loses information
Transform	log1p, sqrt, Box-Cox (positive), Yeo-Johnson (any); fix skew

🔷 Part D: Modelling Prep

Topic	Remember
Imbalance	Don't trust accuracy. Use precision / recall / F1 / PR-AUC. Stratify. Try class weights, SMOTE (training only), threshold tuning
Sampling	Random, systematic, stratified, cluster; bootstrap (with replacement)
Splits	Train (learn) / Validation (tune) / Test (final, once). Stratified, grouped, or chronological as needed. CV for small data
Leakage	Split first; fit transformers on train only; no future info; no duplicates across splits; use pipelines
Pipelines	Pipeline + ColumnTransformer; tune with stepname__param; save with joblib

🔷 Which Scaler / Encoder / Imputer? Fast Table

Situation	Choice
Numeric, skewed, missing	SimpleImputer(median) → PowerTransformer / log1p → StandardScaler
Numeric with outliers	RobustScaler
Nominal, few categories	OneHotEncoder
Nominal, many categories	TargetEncoder (with CV/smoothing), frequency, or group rare values
Ordered categories	OrdinalEncoder(categories=[...])
Target labels	LabelEncoder
Sparse data	MaxAbsScaler

🔷 The Golden Rules

1. SPLIT FIRST, then fit any transformer on TRAIN only.
2. Never touch the test set until the very end.
3. Investigate outliers and missing values before deleting.
4. Keep every learned step inside a PIPELINE.
5. If results look too good to be true, suspect LEAKAGE.

---

✅ Final Takeaways

1. Good data beats fancy models. Collection quality sets the ceiling.
2. EDA and quality checks tell you what to fix before you fix it.
3. Missing values, duplicates and outliers each need a deliberate, justified decision, not an automatic one.
4. Encoding must match the category type (nominal, ordinal, target), and scaling should match the algorithm and the outliers.
5. Imbalanced data needs better metrics and smarter training, not just more accuracy.
6. Splitting correctly (stratified, grouped, or time-based) is what makes your evaluation honest.
7. Data leakage is the silent killer. Pipelines are the cure.

---