← portfolio sign in
↓ .md

Chapter 2 — Essential Data Libraries

Part of Python · page 2 of 4

Level 0+: you've worked through Chapter 1
Time 6–8 hours, best spread over three sessions
You need Your python-data-analysis project folder with its .venv from Chapter 1, and an internet connection for installing libraries
Project milestone The data toolkit installed inside your project, recorded in requirements.txt, and a first Harbour Pharmacy notebook that analyses a month of sales and saves two charts

Where we are

In Chapter 1 you set up an isolated project, then wrote a sales summary using only core Python: a list of dictionaries, a few loops that tallied values into dictionaries, and a lot of careful f-string formatting. It worked, and it took a few dozen lines of careful bookkeeping.

This chapter introduces the libraries that professional analysts use every day. By the end you'll produce that same branch summary in a single line, work with a whole month of data at once, and draw charts. More importantly, you'll understand what each library is for and how they fit together, so later chapters can go deep on each one without you feeling lost.

The four libraries at the heart of this chapter, and how they stack:

            ┌──────────────────────────────┐
            │  Seaborn: statistical charts │
            ├──────────────────────────────┤
            │  Matplotlib: all charts      │
            ├──────────────────────────────┤
            │  pandas: tables (DataFrames) │
            ├──────────────────────────────┤
            │  NumPy: fast arrays of numbers
            └──────────────────────────────┘
                       Python

Each layer is built on the ones beneath it. pandas stores its columns using NumPy; Seaborn draws its charts using Matplotlib. You'll spend most of your working life in the pandas layer, and knowing what sits underneath explains a great deal of pandas' behaviour.


2.1 Libraries, packages and modules: borrowing other people's work

The idea

Nobody writes everything from scratch. A library is a collection of ready-made code, written and tested by other people, that you can pull into your own programs. Python's vocabulary has three overlapping words:

  • A module is a single .py file of Python code that you can import.
  • A package is a folder of related modules bundled together.
  • Library is the everyday, informal word for a package (or group of packages) that gives you a toolkit, such as "the pandas library".

Libraries come from two places:

  1. The standard library ships with Python itself, so it's already on your computer. Useful members include math, statistics, csv, datetime, pathlib and random.
  2. Third-party libraries are published by the community on the Python Package Index (PyPI), at pypi.org, which hosts hundreds of thousands of projects. The installer pip downloads them from PyPI.

Worked example 2.1: using the standard library

The statistics module can describe the week of sales from Chapter 1 without any installing:

import statistics

daily_sales = [412.50, 389.90, 455.10, 501.75, 610.20, 720.40, 298.00]

print(statistics.mean(daily_sales))
print(statistics.median(daily_sales))
print(round(statistics.stdev(daily_sales), 2))

Output:

483.97857142857146
455.1
142.32
  • import statistics makes the module available. You then reach its tools with a dot: statistics.mean(...), just as you reached a string's methods with a dot in Chapter 1.
  • The median is the middle value once the list is sorted. It's often more representative than the mean (average) when a few values are unusually large or small.
  • The standard deviation (stdev) measures how spread out the values are: roughly, how far a typical day sits from the average. Chapter 10 covers it properly.

Three ways to import

import statistics                  # use as statistics.mean(...)
import statistics as st            # a nickname: st.mean(...)
from statistics import mean        # bring one tool in directly: mean(...)

The data world has nicknames everyone uses. You'll see them in every tutorial, book and workplace, so use them too; they make your code instantly readable to other analysts:

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns

Under the hood: how import finds a library

When you write import pandas, Python searches a list of folders for something called pandas, in order, and uses the first match. Sound familiar? It's the same "first match wins" idea as PATH from Chapter 1, applied to libraries. The list lives in a variable called sys.path.

While your virtual environment is active, that list includes your project's private shelf, .venv/…/site-packages, and leaves out the global one. That's how isolation works for libraries. It also explains the single most common beginner error in this chapter:

ModuleNotFoundError: No module named 'pandas'

Nine times out of ten this means one of two things: the library isn't installed in this environment, or the environment isn't active (or your editor is pointing at a different Python). Check for (.venv) in your terminal prompt, or the selected interpreter in VS Code, before anything else.


2.2 Installing the data toolkit into your project

Step 1: activate your environment

Open a terminal, move into your project folder and activate .venv (section 1.2.4). Confirm you see (.venv) in the prompt. Everything in this section assumes the environment is active; otherwise you'd be installing into global Python, which is exactly what we set out to avoid.

Step 2: update pip, then install

python -m pip install --upgrade pip
python -m pip install numpy pandas pyarrow matplotlib seaborn jupyterlab openpyxl

The first line updates pip itself; newer versions handle newer package formats and give clearer error messages. The second line installs seven libraries at once. It may take a minute or two, and you'll see a long list of other packages being installed too; the next section explains why.

What each one is for:

Library Job First used
numpy Fast arrays of numbers: the engine underneath pandas This chapter
pandas Tables (DataFrames): the analyst's main workbench This chapter
pyarrow A high-speed data engine that pandas uses for text columns and file formats. The pandas team strongly recommends installing it alongside pandas 3 Behind the scenes
matplotlib The foundational charting library This chapter
seaborn Statistical charts with good defaults, built on Matplotlib This chapter
jupyterlab The notebook application (section 2.3) This chapter
openpyxl Lets pandas read and write Excel .xlsx files Chapter 3

Why python -m pip instead of plain pip?

python -m pip means "run the pip that belongs to this python". Because your active environment puts .venv first on PATH, python is the environment's interpreter, so its pip installs onto the environment's shelf. Plain pip usually lands in the same place, but on machines with several Pythons installed it can quietly pick a different one. python -m pip removes the guesswork, and the official Python packaging guide uses it for that reason.

A safety net: make pip refuse to install globally

You can tell pip to refuse to install anything unless a virtual environment is active. Run this once (from any terminal):

python -m pip config set global.require-virtualenv true

This writes a setting into your personal pip configuration file (the command prints the file's location as it writes), so it applies to every Python on your account. From then on, an install attempted with no environment active stops immediately with:

ERROR: Could not find an activated virtualenv (required).

That one line guarantees your libraries only ever land inside a project's .venv, which is exactly the isolation Chapter 1 set up. (In the setting's name, global refers to pip's settings section, and has nothing to do with global Python.)

Under the hood: what pip just did

  1. Resolved dependencies. Libraries rely on other libraries. pandas needs NumPy and a date-handling package called python-dateutil; JupyterLab needs dozens of helpers. pip worked out the full list and picked versions that are compatible with each other.
  2. Downloaded wheels. A wheel (a .whl file) is a pre-built, ready-to-install package. NumPy and pandas contain compiled code (written in C for speed), and wheels mean you don't need a compiler on your machine.
  3. Unpacked everything into .venv's site-packages folder. Your global Python is untouched.

Step 3: check the installation

Create ch02/check_versions.py:

import sys

import matplotlib
import numpy as np
import pandas as pd
import seaborn as sns

print("Python     ", sys.version.split()[0])
print("NumPy      ", np.__version__)
print("pandas     ", pd.__version__)
print("Matplotlib ", matplotlib.__version__)
print("Seaborn    ", sns.__version__)

Run it with python ch02/check_versions.py. At the time of writing, the output was:

Output:

Python      3.14.7
NumPy       2.5.3
pandas      3.0.6
Matplotlib  3.11.2
Seaborn     0.13.2

Your version numbers may be newer, and that's fine. What matters for this course is pandas 3.x and NumPy 2.x. If you see pandas 2.something, run python -m pip install --upgrade pandas.

Every library exposes its version as __version__ (two underscores each side). You'll use this when a tutorial behaves differently from what you see: very often, the tutorial was written for an older version.

Step 4: record what you installed in requirements.txt

Your environment is disposable (Chapter 1's house rule 2), so you need a record of what's in it. pip can write that record for you:

python -m pip freeze > requirements.txt

pip freeze lists every installed package with its exact version, and > sends that list into a file instead of the screen. Open requirements.txt and you'll see around a hundred lines like:

numpy==2.5.3
openpyxl==3.1.5
pandas==3.0.6
...

The == pins each package to an exact version. Anyone (including future you on a new laptop) can rebuild an identical environment with:

python -m pip install -r requirements.txt

This small file is what makes your analysis reproducible: the same code, run against the same library versions, produces the same results. Update it whenever you install something new.

The uv way: if you chose uv in Chapter 1, uv add numpy pandas pyarrow matplotlib seaborn jupyterlab openpyxl installs everything and records it in pyproject.toml and a lock file (uv.lock) automatically, with no separate freeze step.


2.3 Jupyter notebooks: your analysis lab bench

The idea

A notebook is a document that mixes three things on one page:

  • code cells, which you run one at a time,
  • the output of each cell (numbers, tables, charts) displayed directly underneath it, and
  • Markdown cells: formatted text for headings, explanations and findings.

Notebooks are saved as .ipynb files. Behind every open notebook runs a kernel: a live Python process that executes your cells and remembers every variable you create, for as long as it keeps running. That memory is what lets cell 5 use a table you loaded in cell 2.

Analysts love notebooks because they show each step and its result side by side, which suits the ask-a-question, look-at-the-answer, ask-another rhythm of analysis. They also make excellent reports: code, charts and written conclusions in one place.

Opening a notebook: two routes

Route A: JupyterLab in your browser

jupyter lab

Run this from your project folder with the environment active. A browser tab opens showing your project's files. Double-click the ch02 folder, then click Notebook → Python 3 (ipykernel) in the Launcher. Rename the new file (right-click → Rename) to ch02_libraries.ipynb. Because you started JupyterLab from inside the environment, the notebook's kernel uses your environment's Python and libraries. To stop JupyterLab, save your work, close the tab and press Ctrl+C in the terminal.

Route B: inside VS Code

  1. In VS Code's Explorer, create a new file ch02/ch02_libraries.ipynb. VS Code opens it as a notebook.
  2. Click Select Kernel (top right) → Python Environments → choose the one marked .venv.
  3. If VS Code offers to install ipykernel, accept. (It came with JupyterLab, so usually it won't need to.)

Either route works. Many people use VS Code day to day and JupyterLab when they want a full-screen analysis space.

Confirm the notebook is using your environment

In the first cell, type this and press Shift+Enter:

import sys
sys.prefix

The result should end in python-data-analysis/.venv (with backslashes on Windows). If it points somewhere else, select the .venv kernel again.

The keys you'll use constantly

Action Keys
Run the cell and move to the next Shift+Enter
Run the cell and stay put Ctrl+Enter
Leave typing mode ("command mode") Esc
Enter typing mode ("edit mode") Enter
Insert a cell above / below (command mode) A / B
Turn a cell into Markdown / code (command mode) M / Y
Delete a cell (command mode) D then D

Notebooks show the last value automatically

Like the interactive shell from Chapter 1, a notebook cell displays the value of its last line without needing print(). For pandas tables this display is especially nice: a neatly formatted table. From here on, examples are written as notebook cells, so you'll often see a bare variable name on the last line; that's what triggers the display.

In this chapter, tables are shown as plain text so they fit in these notes. In a notebook they appear as tidy formatted tables with the same content.

The big gotcha: hidden state

Each cell has a number in square brackets, like [3], showing the order cells actually ran in. You can run cells in any order, re-run them, and delete them, and the kernel still remembers whatever those cells created. So a notebook can look fine on screen while depending on a variable from a cell you deleted an hour ago. Think of it like a lab bench still carrying residue from the last experiment.

The cure is a habit: before trusting or sharing results, use Restart Kernel and Run All Cells (in JupyterLab's Kernel menu; in VS Code, the Restart button then Run All). If everything runs top to bottom without errors, your notebook is honest.


2.4 NumPy: fast numbers in bulk

The idea

NumPy (short for Numerical Python) gives Python one new, extremely important type: the array, officially called an ndarray ("n-dimensional array"). An array is a grid of values that are all the same type, stored side by side in one tightly packed block of memory.

Why does that matter? Watch what a regular list does with maths:

prices_list = [4.50, 6.90, 18.99]
prices_list * 2

Output:

[4.5, 6.9, 18.99, 4.5, 6.9, 18.99]

A list multiplied by 2 repeats itself, just like text did in Chapter 1. To double each price you'd need a loop. Now the NumPy version:

import numpy as np

prices = np.array([4.50, 6.90, 18.99])
prices * 2

Output:

array([ 9.  , 13.8 , 37.98])

The array applied * 2 to every element. This element-by-element behaviour is called vectorisation, and it's the single most important idea in this chapter. pandas columns behave the same way.

Under the hood: why arrays are fast

A Python list is a row of pointers to separate objects scattered around memory, each carrying its own type information. To double every item, Python must visit each object, check its type, do the maths and build a new object, one at a time.

An array stores raw numbers side by side, all the same type, so NumPy can hand the whole block to a tight loop written in the C language, which runs at the full speed of your processor. On a typical laptop, doubling a million numbers looks something like this:

big_list = list(range(1_000_000))
big_array = np.arange(1_000_000)

%timeit [x * 2 for x in big_list]    # roughly 40 ms
%timeit big_array * 2                 # roughly 0.7 ms: about 60 times faster

%timeit is a notebook-only helper (a "magic command") that runs a line many times and reports the typical duration. Your numbers will differ, and the gap is what matters. With the data sizes analysts handle, that gap is the difference between waiting a second and waiting a minute.

Creating arrays

week = np.array([412.50, 389.90, 455.10, 501.75, 610.20, 720.40, 298.00])
day_numbers = np.arange(1, 8)            # like range(): 1 to 7, stop excluded
evenly_spaced = np.linspace(0, 1, 5)     # 5 evenly spaced values from 0 to 1
empty_tally = np.zeros(4)                # four zeros

print(day_numbers)
print(evenly_spaced)
print(empty_tally)

Output:

[1 2 3 4 5 6 7]
[0.   0.25 0.5  0.75 1.  ]
[0. 0. 0. 0.]

Every array knows a few facts about itself:

print(week.shape)   # size along each dimension
print(week.ndim)    # number of dimensions
print(week.size)    # total number of elements
print(week.dtype)   # the single data type shared by every element

Output:

(7,)
1
7
float64

(7,) reads as "7 elements along one dimension". The trailing comma is how Python writes a tuple with one item. float64 means "64-bit floating-point number", the standard decimal type; int64 is its whole-number cousin. You'll see these dtype names constantly in pandas.

Worked example 2.2: a week of sales, vectorised

costs = np.array([250.00, 240.10, 270.30, 300.00, 350.60, 410.00, 190.50])

margin = week - costs              # element by element
margin_pct = margin / week * 100   # still element by element

print(margin)
print(margin_pct.round(1))

Output:

[162.5  149.8  184.8  201.75 259.6  310.4  107.5 ]
[39.4 38.4 40.6 40.2 42.5 43.1 36.1]

Two arrays of the same length combine position by position: Monday's sales minus Monday's costs, Tuesday's minus Tuesday's, and so on. No loop required.

Summarising an array

print(week.sum())
print(week.mean())
print(week.max())
print(week.argmax())   # the POSITION of the largest value
print(week.std().round(2))

Output:

3387.85
483.9785714285714
720.4
5
131.76

argmax() answers a different question from max(): where is the biggest value? Position 5 is the sixth day, Saturday, since positions start at 0.

The standard deviation here differs slightly from the statistics.stdev result in section 2.1 (131.76 vs 142.32). NumPy's std() treats the numbers as a whole population by default, while statistics.stdev() treats them as a sample from a larger population. Chapter 10 explains when to use which; for now, just know the two exist.

A display quirk you'll meet in notebooks

Type a summary as the last line of a cell and the notebook shows something slightly surprising:

week.mean()

Output:

np.float64(483.9785714285714)

Since NumPy 2, displaying a single NumPy number also shows its type: np.float64(...) means "a NumPy 64-bit float with this value". It behaves exactly like a normal Python float. print() shows the bare number, as you saw above, and float(week.mean()) converts it to a plain Python float. Older tutorials, written before NumPy 2, show only the bare number.

Boolean masks: the most useful pattern in data analysis

Comparing an array to a value produces an array of True/False, one per element:

target = 450
week > target

Output:

array([False, False,  True,  True,  True,  True, False])

That True/False array is called a boolean mask. It has two superpowers.

1. Use it to select only the matching elements:

week[week > target]

Output:

array([455.1 , 501.75, 610.2 , 720.4 ])

Read week[week > target] as "the values of week, where week is above the target". Only positions marked True survive.

2. Count or measure with it. In arithmetic, True counts as 1 and False as 0:

print((week > target).sum())    # how many days beat the target
print((week > target).mean())   # what fraction of days beat the target

Output:

4
0.5714285714285714

Four days out of seven, or about 57%. The same "sum a mask to count, average it to get a proportion" trick works in pandas, and you'll use it weekly.

Combining conditions

Use & for "and", | for "or", and ~ for "not". Wrap each condition in brackets:

weekend = np.array([False, False, False, False, False, True, True])

week[(week > target) & ~weekend]    # strong weekdays

Output:

array([455.1 , 501.75, 610.2 ])

Why the brackets? In Python, & is worked out before >, so without brackets Python would try to calculate target & ~weekend first and fail. Brackets make the order explicit. (The words and/or from Chapter 1 work only on single True/False values; for arrays you need the symbols.)

Choosing between two values: np.where

np.where(condition, value_if_true, value_if_false) builds a new array element by element:

np.where(week > target, "Above", "Below")

Output:

array(['Below', 'Below', 'Above', 'Above', 'Above', 'Above', 'Below'],
      dtype='<U5')

This is Python's version of Excel's IF() applied to a whole column at once. You'll use it for labelling in Chapter 5. (The dtype='<U5' at the end is NumPy's own text type: Unicode text up to 5 characters long. In pandas you'll see text as str instead.)

Two dimensions: rows and columns

Arrays can have more than one dimension. A two-dimensional array is a table of numbers. Here are four branches (rows) across seven days (columns):

branch_week = np.array([
    [520, 480, 610, 590, 700, 820, 400],   # Lambton Quay
    [410, 395, 450, 470, 600, 760, 380],   # Cuba Street
    [300, 310, 290, 350, 420, 510, 330],   # Petone
    [220, 230, 260, 240, 300, 380, 250],   # Karori
])
print(branch_week.shape)
print(branch_week[0, 5])     # row 0 (Lambton Quay), column 5 (Saturday)
print(branch_week[2])        # the whole of row 2 (Petone)
print(branch_week[:, 0])     # every row, column 0 (Monday)

Output:

(4, 7)
820
[300 310 290 350 420 510 330]
[520 410 300 220]

Indexing takes [row, column]. A lone : means "everything along this dimension", so [:, 0] is "all rows, first column".

Understanding axis

Summaries on a 2-D array take an axis setting, and it trips everyone up at first. The trick: axis names the direction that gets collapsed.

                 axis=1 → collapse ACROSS each row
               ┌─────────────────────────────────┐
  axis=0       │ 520  480  610  590  700  820 400 │ → one total per row (per branch)
  ↓            │ 410  395  450  470  600  760 380 │
  collapse     │ 300  310  290  350  420  510 330 │
  DOWN each    │ 220  230  260  240  300  380 250 │
  column       └─────────────────────────────────┘
               ↓ one total per column (per day)
print(branch_week.sum(axis=1))   # per branch: add across each row
print(branch_week.sum(axis=0))   # per day: add down each column

Output:

[4120 3465 2510 1880]
[1450 1415 1610 1650 2020 2470 1360]

pandas uses the same axis idea, so this picture will keep paying off.

Missing values: np.nan

Real data has gaps. NumPy marks a missing number with np.nan ("Not a Number"), a special floating-point value. It's contagious: any arithmetic involving it produces nan.

with_gap = np.array([412.50, np.nan, 455.10])

print(with_gap.sum())
print(np.nansum(with_gap))
print(np.nanmean(with_gap))

Output:

nan
867.6
433.8

np.nansum and np.nanmean skip the gaps. Because nan is a float, a column of whole numbers with even one gap becomes a float column. You'll see exactly this happen in Chapter 3, and pandas handles missing values more gracefully than raw NumPy (its summaries skip them by default).

Random numbers, reproducibly

Analysts often need realistic practice data, or random samples. NumPy's random generator does this:

rng = np.random.default_rng(seed=42)

print(rng.integers(1, 5, size=10))          # 10 whole numbers from 1 to 4
print(rng.normal(loc=100, scale=15, size=3).round(1))   # 3 bell-curve values around 100

Output:

[1 4 3 2 2 4 1 3 1 1]
[ 80.5 101.9  95.3]

The seed is a starting point for the generator. The same seed always produces the same "random" numbers, so your results are reproducible: run the cell again, or send it to a colleague, and you'll both get identical values. normal draws from the familiar bell-shaped curve: loc is its centre and scale controls the spread.


2.5 pandas: tables with superpowers

The idea

pandas is the library you'll spend most of your analyst life in. Its name comes from "panel data", an economics term for tables of observations over time. It gives you two structures:

  • a Series: one column of values, each with a label; and
  • a DataFrame: a table made of several Series side by side, all sharing the same row labels.

The row labels are called the index. Think of a DataFrame as a spreadsheet where every column knows its own data type, every row has a label, and every operation can be applied to a whole column at once.

Under the hood, each column is essentially a NumPy array (or, for text, a pyarrow array). So everything you just learnt, including vectorisation, boolean masks, nan and axis, carries straight across.

Series: one labelled column

import pandas as pd

sales = pd.Series(
    [412.50, 389.90, 455.10, 501.75, 610.20, 720.40, 298.00],
    index=["Mon", "Tue", "Wed", "Thu", "Fri", "Sat", "Sun"],
    name="daily_sales",
)
sales

Output:

Mon    412.50
Tue    389.90
Wed    455.10
Thu    501.75
Fri    610.20
Sat    720.40
Sun    298.00
Name: daily_sales, dtype: float64

The left column is the index (labels), the right column holds the values, and the footer shows the Series' name and dtype. Selection works by label, and everything is vectorised:

print(sales["Sat"])
print(sales[sales > 500])

Output:

720.4
Thu    501.75
Fri    610.20
Sat    720.40
Name: daily_sales, dtype: float64

DataFrames: whole tables

You can build a DataFrame from the exact structure you used in Chapter 1: a list of dictionaries, where each dictionary is one row.

Worked example 2.3: Chapter 1's sales, now as a DataFrame

sales_records = [
    {"order_id": 1001, "branch": "Petone", "product": "Vapour Rub 50g",
     "unit_price": 9.99, "quantity": 1, "loyalty_member": True},
    {"order_id": 1002, "branch": "Lambton Quay", "product": "Throat Lozenges 16pk",
     "unit_price": 8.99, "quantity": 2, "loyalty_member": True},
    {"order_id": 1003, "branch": "Karori", "product": "Throat Lozenges 16pk",
     "unit_price": 8.99, "quantity": 4, "loyalty_member": False},
    {"order_id": 1004, "branch": "Karori", "product": "Throat Lozenges 16pk",
     "unit_price": 8.99, "quantity": 1, "loyalty_member": True},
    {"order_id": 1005, "branch": "Petone", "product": "Plasters 40pk",
     "unit_price": 5.99, "quantity": 1, "loyalty_member": True},
    {"order_id": 1006, "branch": "Lambton Quay", "product": "Multivitamin 60tab",
     "unit_price": 24.99, "quantity": 1, "loyalty_member": False},
    {"order_id": 1007, "branch": "Cuba Street", "product": "Ibuprofen 200mg 24pk",
     "unit_price": 6.90, "quantity": 1, "loyalty_member": True},
    {"order_id": 1008, "branch": "Cuba Street", "product": "Cold & Flu Capsules 24pk",
     "unit_price": 14.50, "quantity": 3, "loyalty_member": False},
]

df = pd.DataFrame(sales_records)
df

Output:

   order_id        branch                   product  unit_price  quantity  loyalty_member
0      1001        Petone            Vapour Rub 50g        9.99         1            True
1      1002  Lambton Quay      Throat Lozenges 16pk        8.99         2            True
2      1003        Karori      Throat Lozenges 16pk        8.99         4           False
3      1004        Karori      Throat Lozenges 16pk        8.99         1            True
4      1005        Petone             Plasters 40pk        5.99         1            True
5      1006  Lambton Quay        Multivitamin 60tab       24.99         1           False
6      1007   Cuba Street      Ibuprofen 200mg 24pk        6.90         1            True
7      1008   Cuba Street  Cold & Flu Capsules 24pk       14.50         3           False

The dictionary keys became column names, and pandas added a default index of 0 to 7 down the left. (You can also build a DataFrame from a dictionary of lists, one list per column: pd.DataFrame({"branch": [...], "quantity": [...]}). Both are common.)

The first-look toolkit

Whenever you meet a new DataFrame, these four tell you what you're dealing with:

print(df.shape)       # (rows, columns)
print(df.columns.tolist())

Output:

(8, 6)
['order_id', 'branch', 'product', 'unit_price', 'quantity', 'loyalty_member']
df.dtypes

Output:

order_id            int64
branch                str
product               str
unit_price        float64
quantity            int64
loyalty_member       bool
dtype: object

Each column has one dtype. int64 and float64 are NumPy's number types; bool is True/False; and str marks text.

pandas 3 note: the str dtype for text is new in pandas 3.0, released January 2026. Tutorials and books written for earlier versions show text columns as object. Both mean "this column holds text"; the new str type is stricter (it only ever holds text or missing values) and faster.

df.info()

Output:

<class 'pandas.DataFrame'>
RangeIndex: 8 entries, 0 to 7
Data columns (total 6 columns):
 #   Column          Non-Null Count  Dtype  
---  ------          --------------  -----  
 0   order_id        8 non-null      int64  
 1   branch          8 non-null      str    
 2   product         8 non-null      str    
 3   unit_price      8 non-null      float64
 4   quantity        8 non-null      int64  
 5   loyalty_member  8 non-null      bool   
dtypes: bool(1), float64(1), int64(2), str(2)
memory usage: 679.0 bytes

info() combines shape, column names, dtypes and, crucially, the non-null count: how many values in each column are present (not missing). When a column shows fewer non-null values than there are rows, you've found missing data. All 8 are present here; Chapter 3's real dataset won't be so tidy.

df.describe()

Output:

         order_id  unit_price  quantity
count     8.00000    8.000000  8.000000
mean   1004.50000   11.167500  1.750000
std       2.44949    6.127178  1.164965
min    1001.00000    5.990000  1.000000
25%    1002.75000    8.467500  1.000000
50%    1004.50000    8.990000  1.000000
75%    1006.25000   11.117500  2.250000
max    1008.00000   24.990000  4.000000

describe() summarises every numeric column: count, mean, standard deviation, minimum, the 25th/50th/75th percentiles (the 50% row is the median) and maximum. It's a thirty-second health check. Here it says, for example, that the median sale is 1 unit (the 50% row), the average is 1.75 units, and the largest is 4 units. (The order_id summary is meaningless, since IDs are labels rather than quantities. Ignoring columns like that is part of reading describe() sensibly.)

Selecting columns

df["product"]

Output:

0              Vapour Rub 50g
1        Throat Lozenges 16pk
2        Throat Lozenges 16pk
3        Throat Lozenges 16pk
4               Plasters 40pk
5          Multivitamin 60tab
6        Ibuprofen 200mg 24pk
7    Cold & Flu Capsules 24pk
Name: product, dtype: str

Single brackets with one column name give you a Series. Double brackets give you a DataFrame with just those columns:

df[["branch", "product", "quantity"]]

Output:

         branch                   product  quantity
0        Petone            Vapour Rub 50g         1
1  Lambton Quay      Throat Lozenges 16pk         2
2        Karori      Throat Lozenges 16pk         4
3        Karori      Throat Lozenges 16pk         1
4        Petone             Plasters 40pk         1
5  Lambton Quay        Multivitamin 60tab         1
6   Cuba Street      Ibuprofen 200mg 24pk         1
7   Cuba Street  Cold & Flu Capsules 24pk         3

Why double? The outer brackets mean "select from df", and the inner brackets are a Python list of the columns you want. Chapter 3 covers selection in much more depth.

Creating a column: vectorised, no loop

In Chapter 1 you wrote a line_total() function and called it inside a loop. In pandas:

df["revenue"] = df["unit_price"] * df["quantity"]
df[["product", "unit_price", "quantity", "revenue"]]

Output:

                    product  unit_price  quantity  revenue
0            Vapour Rub 50g        9.99         1     9.99
1      Throat Lozenges 16pk        8.99         2    17.98
2      Throat Lozenges 16pk        8.99         4    35.96
3      Throat Lozenges 16pk        8.99         1     8.99
4             Plasters 40pk        5.99         1     5.99
5        Multivitamin 60tab       24.99         1    24.99
6      Ibuprofen 200mg 24pk        6.90         1     6.90
7  Cold & Flu Capsules 24pk       14.50         3    43.50

One line. pandas multiplied the two columns row by row (vectorisation again) and stored the result as a new column called revenue.

Filtering rows with a boolean mask

Exactly the same pattern as NumPy:

bulk = df["quantity"] >= 3
df[bulk]

Output:

   order_id       branch                   product  unit_price  quantity  loyalty_member  revenue
2      1003       Karori      Throat Lozenges 16pk        8.99         4           False    35.96
7      1008  Cuba Street  Cold & Flu Capsules 24pk       14.50         3           False    43.50

And combined conditions, with &, |, ~ and brackets around each condition:

df[(df["branch"] == "Karori") & df["loyalty_member"]]

Output:

   order_id  branch               product  unit_price  quantity  loyalty_member  revenue
3      1004  Karori  Throat Lozenges 16pk        8.99         1            True     8.99

df["loyalty_member"] is already a column of True/False values, so it can be used as a condition directly.

Sorting and counting

df.sort_values("revenue", ascending=False).head(3)

Output:

   order_id        branch                   product  unit_price  quantity  loyalty_member  revenue
7      1008   Cuba Street  Cold & Flu Capsules 24pk       14.50         3           False    43.50
2      1003        Karori      Throat Lozenges 16pk        8.99         4           False    35.96
5      1006  Lambton Quay        Multivitamin 60tab       24.99         1           False    24.99

sort_values() returns a new, sorted DataFrame, and .head(3) keeps the first three rows. Chaining methods like this (do one thing, then the next) is very common in pandas code.

df["branch"].value_counts()

Output:

branch
Petone          2
Lambton Quay    2
Karori          2
Cuba Street     2
Name: count, dtype: int64

value_counts() counts how often each distinct value appears, most frequent first. It's the pandas version of the counting loop you wrote in Chapter 1. In this small sample every branch appears twice, so the tie is listed in order of first appearance; with real data the differences are usually more interesting.

Worked example 2.4: Chapter 1's summary, in pandas

Here's the payoff. The revenue-by-branch table from your Chapter 1 milestone script took a loop, a dictionary and careful bookkeeping:

df.groupby("branch")["revenue"].sum()

Output:

branch
Cuba Street     50.40
Karori          44.95
Lambton Quay    42.97
Petone          15.98
Name: revenue, dtype: float64

Same numbers as Chapter 1 (Cuba Street 50.40, Karori 44.95, Lambton Quay 42.97, Petone 15.98), in one line. Read it as: group the rows by branch, take the revenue column within each group, and add it up. Chapter 6 is devoted to groupby, because it answers a huge share of business questions.

The rest of the Chapter 1 report takes a line each:

total = df["revenue"].sum()
loyalty_share = df.loc[df["loyalty_member"], "revenue"].sum() / total
best_seller = df.groupby("product")["quantity"].sum().idxmax()

print(f"Total revenue: ${total:,.2f}")
print(f"Loyalty share: {loyalty_share:.1%}")
print(f"Best seller:   {best_seller}")

Output:

Total revenue: $154.30
Loyalty share: 32.3%
Best seller:   Throat Lozenges 16pk

Two new tools here:

  • df.loc[rows, column] selects by row condition and column name at the same time. Here: the rows where loyalty_member is True, and only the revenue column. Chapter 3 explains .loc fully.
  • idxmax() returns the label of the largest value, the pandas cousin of NumPy's argmax().

Changing values safely with .loc

To change some values in a column, based on a condition, use .loc with the condition and the column name together:

df["discount"] = 0.0
df.loc[df["loyalty_member"], "discount"] = 0.05
df[["order_id", "loyalty_member", "discount"]]

Output:

   order_id  loyalty_member  discount
0      1001            True      0.05
1      1002            True      0.05
2      1003           False      0.00
3      1004            True      0.05
4      1005            True      0.05
5      1006           False      0.00
6      1007            True      0.05
7      1008           False      0.00

Read it as: in the rows where the customer is a loyalty member, set discount to 0.05.

pandas 3 note: always assign with a single .loc. Older tutorials sometimes write df[df["loyalty_member"]]["discount"] = 0.05, two sets of brackets one after another (called chained assignment). In pandas 3 that line never changes df. It changes a temporary copy that's immediately thrown away, and pandas prints a ChainedAssignmentError warning to tell you so. This comes from a pandas 3 design called Copy-on-Write: every selection behaves like an independent copy, which also means the old SettingWithCopyWarning you'll see complained about online has been removed. The rule to remember is simple: to change values in place, use one .loc[rows, column] = value.

Missing values in pandas

pandas shows missing values as NaN (or None/<NA> in some dtypes), and, helpfully, its summaries skip them automatically:

quantities = pd.Series([1, 2, None, 4])
print(quantities)
print("Sum:", quantities.sum())
print("Missing:", quantities.isna().sum())

Output:

0    1.0
1    2.0
2    NaN
3    4.0
dtype: float64
Sum: 7.0
Missing: 1

Two things to notice. First, the column became float64 because of the gap, just as you saw with NumPy. Second, .isna() makes a boolean mask of "is this missing?", and .sum() counts the Trues: the same mask-and-count trick from earlier. Chapter 4 covers deciding what to do about missing values.


2.6 Matplotlib: drawing charts

The idea

Matplotlib is Python's foundational charting library. Nearly every other Python charting tool, including Seaborn and pandas' own .plot(), draws through it.

Two words form the mental model:

  • A Figure is the whole canvas: the image you'd save or paste into a report.
  • An Axes is one chart inside that figure: the plotting area together with its x-axis, y-axis, title and labels. A figure can hold one Axes or several.

"Axes" (the chart) and "axis" (one of the lines with tick marks) are different things, confusingly close in spelling. You'll mostly talk to the Axes.

The Matplotlib documentation recommends this pattern, and it's the one we'll use throughout the course:

fig, ax = plt.subplots()     # 1. make a Figure with one Axes
ax.plot(...)                 # 2. draw on the Axes
ax.set_title(...)            # 3. label it
fig.savefig(...)             # 4. save it (optional)
plt.show()                   # 5. display it

You'll also see shorter code in tutorials that calls plt.plot(...) directly, relying on Matplotlib to keep track of "the current chart" behind the scenes. That shortcut works for quick one-off charts. The fig, ax style is more explicit, and it scales cleanly when you need two or more charts side by side, so it's worth learning first.

Worked example 2.5: a line chart of the week

import matplotlib.pyplot as plt

days = ["Mon", "Tue", "Wed", "Thu", "Fri", "Sat", "Sun"]

fig, ax = plt.subplots(figsize=(8, 4))
ax.plot(days, week, marker="o", label="Daily sales")
ax.axhline(target, color="grey", linestyle="--", label="Target ($450)")
ax.set_title("Harbour Pharmacy: sales by day")
ax.set_xlabel("Day")
ax.set_ylabel("Sales (NZD)")
ax.legend()

fig.savefig("../output/ch02_week_line.png", dpi=150, bbox_inches="tight")
plt.show()

What each line does:

Code Effect
plt.subplots(figsize=(8, 4)) a figure 8 inches wide and 4 tall, containing one Axes
ax.plot(days, week, marker="o") a line through the seven points, with a dot on each
label="..." the name to show in the legend
ax.axhline(target, ...) a horizontal reference line across the chart at the target
set_title, set_xlabel, set_ylabel the chart title and axis labels
ax.legend() draw the legend, using the labels given above
fig.savefig(...) save as a PNG image in your output/ folder; dpi=150 sets the sharpness and bbox_inches="tight" trims empty margins
plt.show() display the chart

You'll see a line rising from Monday to a Saturday peak, then dropping sharply on Sunday, with a dashed grey target line that four of the dots sit above.

The ../output/ path means "go up one folder from where this notebook lives (ch02/), then into output/". Chapter 3 explains file paths properly.

Why end chart cells with plt.show()? In a notebook the chart appears anyway, and without plt.show() the cell also prints the value of its last line: something like <matplotlib.legend.Legend at 0x7f…>. It's harmless, just noisy. In a .py script, plt.show() is what actually opens the chart window.

Worked example 2.6: a bar chart from a pandas result

Charts usually come straight from pandas results. Let's chart the four branches' weekly totals from the 2-D array:

branch_names = ["Lambton Quay", "Cuba Street", "Petone", "Karori"]
branch_totals = pd.Series(branch_week.sum(axis=1), index=branch_names)

fig, ax = plt.subplots(figsize=(7, 4))
ax.bar(branch_totals.index, branch_totals.values, color="steelblue")
ax.set_title("Weekly sales by branch")
ax.set_ylabel("Sales (NZD)")
plt.show()

ax.bar(x_positions, heights) draws one bar per branch. .index gives the Series' labels and .values gives its numbers.

pandas also has a .plot() shortcut that does the same job and fills in some labels for you:

fig, ax = plt.subplots(figsize=(7, 4))
branch_totals.sort_values().plot(kind="barh", ax=ax, color="steelblue")
ax.set_title("Weekly sales by branch")
ax.set_xlabel("Sales (NZD)")
plt.show()

kind="barh" makes a horizontal bar chart, which is often easier to read when the category names are long, and sorting first puts the bars in order. Passing ax=ax tells pandas to draw on your Axes, so you keep full control over the labels.

Worked example 2.7: a histogram, and two charts side by side

A histogram shows how values are distributed: it splits the range into "bins" and counts how many values fall in each. Let's simulate 500 transaction values and look at their shape, next to a second chart:

rng = np.random.default_rng(seed=1)
transaction_values = rng.gamma(shape=2.0, scale=12.0, size=500).round(2)

fig, axes = plt.subplots(1, 2, figsize=(11, 4))

axes[0].hist(transaction_values, bins=30, color="teal", edgecolor="white")
axes[0].set_title("Transaction values (500 simulated sales)")
axes[0].set_xlabel("Value (NZD)")
axes[0].set_ylabel("Number of transactions")

axes[1].plot(days, week.cumsum(), marker="o")
axes[1].set_title("Running total across the week")
axes[1].set_ylabel("Cumulative sales (NZD)")

fig.tight_layout()
plt.show()
  • plt.subplots(1, 2) makes 1 row and 2 columns of Axes, returned as an array called axes. axes[0] is the left chart and axes[1] the right.
  • rng.gamma(...) produces skewed, all-positive values, a realistic shape for purchase amounts: most transactions are small, with a long tail of bigger ones. The histogram shows a hump near $10–$20 and a tail stretching to the right.
  • .cumsum() is a cumulative sum, a running total.
  • fig.tight_layout() nudges the charts apart so their labels don't overlap.

Chapter 9 is dedicated to choosing and polishing charts. For now, the pattern fig, ax = plt.subplots() → draw → label → show is the one to have in your fingers.


2.7 Seaborn: statistical charts with sensible defaults

The idea

Seaborn is built on Matplotlib and designed to work directly with pandas DataFrames. You tell it which columns to use for x, y and colour, and it handles the grouping, summarising and legends for you. Its defaults are also more attractive than Matplotlib's.

import seaborn as sns

sns.set_theme(style="whitegrid")   # a clean look for every chart from now on

Worked example 2.8: revenue by branch, split by loyalty

fig, ax = plt.subplots(figsize=(8, 4))
sns.barplot(
    data=df,
    x="branch",
    y="revenue",
    hue="loyalty_member",
    estimator="sum",
    errorbar=None,
    ax=ax,
)
ax.set_title("Revenue by branch and loyalty status")
ax.set_ylabel("Revenue (NZD)")
plt.show()
Setting Meaning
data=df the DataFrame to draw from
x="branch", y="revenue" the columns for each axis
hue="loyalty_member" split each bar into colours by this column, with a legend added automatically
estimator="sum" add up the revenue within each group
errorbar=None don't draw error bars
ax=ax draw on our Axes

An important Seaborn default to know: barplot shows the mean of each group, with a black error bar showing a 95% confidence interval (a range that expresses uncertainty about that mean; Chapter 12 explains them). That's a sensible default for scientific data and a frequent trap for business reporting, where people usually expect bar heights to be totals. Setting estimator="sum" and errorbar=None gives you the plain totals most stakeholders expect.

Worked example 2.9: a distribution in one line

fig, ax = plt.subplots(figsize=(7, 4))
sns.histplot(x=transaction_values, bins=30, kde=True, ax=ax)
ax.set_title("Transaction values")
ax.set_xlabel("Value (NZD)")
plt.show()

kde=True adds a smooth curve over the bars (a kernel density estimate: a smoothed outline of the histogram's shape), which makes the overall pattern easier to see.

As a rule of thumb: reach for Seaborn for quick, good-looking charts built straight from a DataFrame, especially when you want to split by category. Reach for Matplotlib directly when you need fine control over every detail. Because Seaborn draws on a Matplotlib Axes, you can always mix the two, as you just did with ax.set_title().


2.8 The wider toolbox

You won't install these yet. They're here so the names feel familiar when they appear:

Library What it does Where it appears
openpyxl Read and write Excel files (already installed) Chapter 3, Chapter 13
sqlite3 (standard library), SQLAlchemy Talk to databases using SQL Chapter 11
SciPy Scientific functions, including statistical tests Chapter 12
statsmodels Statistical models and tests with detailed reports Chapter 12
Plotly, Streamlit Interactive charts and simple web dashboards Chapter 14
requests Fetch data from websites and APIs Later chapters, as needed
scikit-learn Machine learning: prediction and classification A natural next step after this course
Polars A newer, very fast DataFrame library; an alternative to pandas Worth a look once pandas feels comfortable

2.9 Project milestone: Harbour Pharmacy's first notebook

Time to put the toolkit together. The operations manager wants to see a whole month at a glance: how each branch performed in total, their best days, and how often each branch cleared a $1,000-a-day target.

You don't have the real month's data yet (that arrives in Chapter 3), so you'll simulate a realistic month with NumPy. Creating plausible fake data is a genuine analyst skill: it lets you build and test an analysis before the real data turns up.

Create a new notebook, ch02/ch02_milestone.ipynb. Put each block below in its own cell, and add a Markdown cell above each one with a short heading describing what it does (for example, ## 1. Simulate the month).

Cell 1: imports and settings

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import seaborn as sns

sns.set_theme(style="whitegrid")

Cell 2: simulate 28 days of sales for four branches

rng = np.random.default_rng(seed=7)

dates = pd.date_range("2026-03-02", periods=28, freq="D")
typical_day = {"Lambton Quay": 1450, "Cuba Street": 1150, "Petone": 900, "Karori": 700}

daily = pd.DataFrame(index=dates)
for branch, typical in typical_day.items():
    daily[branch] = rng.normal(loc=typical, scale=typical * 0.15, size=len(dates)).round(2)

daily.index.name = "date"
daily.head()

Output:

            Lambton Quay  Cuba Street   Petone  Karori
date                                                  
2026-03-02       1450.27      1141.63   891.02  679.29
2026-03-03       1514.98      1169.55   990.08  583.02
2026-03-04       1390.38       886.05  1094.20  698.79
2026-03-05       1256.30      1067.59   808.79  653.42
2026-03-06       1351.11       981.21   927.42  822.44

What's happening:

  • pd.date_range("2026-03-02", periods=28, freq="D") creates 28 consecutive dates starting on Monday 2 March. (Dates get a whole chapter, Chapter 8.)
  • pd.DataFrame(index=dates) makes an empty table whose row labels are those dates.
  • The loop adds one column per branch. Each column is 28 bell-curve values centred on that branch's typical day, with a spread of 15%. The fixed seed means you'll get exactly the numbers shown here.

Cell 3: headline numbers

monthly_totals = daily.sum().sort_values(ascending=False)
monthly_totals.round(2)

Output:

Lambton Quay    37884.11
Cuba Street     31952.05
Petone          25107.09
Karori          19453.71
dtype: float64

daily.sum() adds down each column (pandas' default is axis=0, the "collapse down" direction you met with NumPy), giving one total per branch.

Cell 4: each branch's best day, and days on target

target = 1000

summary = pd.DataFrame({
    "total": daily.sum().round(2),
    "daily_average": daily.mean().round(2),
    "best_day": daily.idxmax().dt.strftime("%a %d %b"),
    "days_on_target": (daily >= target).sum(),
})
summary.sort_values("total", ascending=False)

Output:

                 total  daily_average    best_day  days_on_target
Lambton Quay  37884.11        1353.00  Mon 09 Mar              27
Cuba Street   31952.05        1141.14  Mon 23 Mar              23
Petone        25107.09         896.68  Wed 25 Mar               5
Karori        19453.71         694.78  Sat 21 Mar               0

Take this one slowly; it combines several ideas:

  • Building a DataFrame from a dictionary of Series: each key becomes a column, and pandas lines the rows up by their shared labels (the branch names).
  • daily.idxmax() finds the date label of each column's largest value. .dt.strftime("%a %d %b") formats those dates as text such as Sat 21 Mar. (%a = short weekday, %d = day number, %b = short month; Chapter 8 lists the rest.)
  • (daily >= target) compares the whole table to 1,000 at once, producing a table of True/False. .sum() then counts the Trues in each column: the NumPy mask-counting trick, applied to every branch at once.

Notice Karori shows zero days on target. With a typical day of $700, a single $1,000 target is unfair on smaller branches, which is exactly the kind of observation an analyst raises. (Exercise 2.8 asks you to fix it.)

Cell 5: charts, saved for the report

fig, axes = plt.subplots(1, 2, figsize=(13, 4.5))

for branch in daily.columns:
    axes[0].plot(daily.index, daily[branch], label=branch, linewidth=1.5)
axes[0].axhline(target, color="grey", linestyle="--", linewidth=1)
axes[0].set_title("Daily sales by branch, March 2026 (simulated)")
axes[0].set_ylabel("Sales (NZD)")
axes[0].tick_params(axis="x", labelrotation=45)
axes[0].legend(ncols=2, fontsize=9)

monthly_totals.sort_values().plot(kind="barh", ax=axes[1], color="steelblue")
axes[1].set_title("Total sales over 28 days")
axes[1].set_xlabel("Sales (NZD)")
axes[1].set_ylabel("")

fig.tight_layout()
fig.savefig("../output/ch02_milestone.png", dpi=150, bbox_inches="tight")
plt.show()

The left chart loops over the branch columns and draws one line for each, labelled with the branch name so the legend knows what to call it. Matplotlib recognises that the x values are dates and spaces the date labels sensibly; tick_params(axis="x", labelrotation=45) tilts them so they don't collide. (pandas can draw all four lines in one go with daily.plot(ax=axes[0]), and it's handy for a quick look; on daily dates its automatic labels can get crowded, which is why we're drawing the lines ourselves here.)

What you'll see: on the left, four wiggly lines in clear bands, with Lambton Quay on top and Karori at the bottom, and the dashed target line running between Cuba Street and Petone. On the right, four horizontal bars ranked by total.

Cell 6 (Markdown): findings

End the notebook with a Markdown cell, written in plain language for the operations manager. For example:

## Findings (simulated data)

- Lambton Quay generated the most revenue over the 28 days, about $37.9k,
  nearly double Karori's $19.5k.
- Lambton Quay beat the $1,000 daily target on 27 of 28 days; Karori never did.
- A single company-wide target doesn't suit branches of such different sizes.
  Next step: set targets relative to each branch's typical day.

Finally, Restart Kernel and Run All to prove the notebook runs cleanly from top to bottom. Check that output/ now contains ch02_week_line.png and ch02_milestone.png.

Writing findings in plain words at the end of an analysis is a habit worth starting now. Stakeholders read the conclusions; the code is how you earn the right to state them.


2.10 Exercises

Work in a notebook, ch02/ch02_exercises.ipynb, with a Markdown heading for each exercise. Attempt each before opening the solution.

Available this chapter: 250 XP.

Warm-up (10 XP each)

Exercise 2.1: Environment health check. In a notebook cell, print sys.prefix and the pandas and NumPy versions. Then, in the terminal, regenerate requirements.txt and confirm pandas== appears in it.

Show a solution ```python import sys import numpy as np import pandas as pd print(sys.prefix) print("pandas", pd.__version__) print("NumPy", np.__version__) ``` The path should end in `.venv`. Then, in the terminal with the environment active: ```bash python -m pip freeze > requirements.txt ``` Open `requirements.txt` in VS Code and search for `pandas==` (`Ctrl+F`).

Exercise 2.2: Stock check with NumPy. Create an array of stock counts, stock = np.array([120, 9, 0, 31, 18, 64, 3]), and a reorder point of 25. Using boolean masks, print (a) how many products need reordering (below 25, including zero), and (b) the stock counts of those products.

Show a solution ```python import numpy as np stock = np.array([120, 9, 0, 31, 18, 64, 3]) reorder_point = 25 needs_reorder = stock < reorder_point print(needs_reorder.sum()) print(stock[needs_reorder]) ``` **Output:** ```text 4 [ 9 0 18 3] ```

Exercise 2.3: A price list Series. Create a Series of these prices, indexed by product name: Paracetamol 500mg 20pk 4.50, Vitamin C 1000mg 60tab 18.99, Sunscreen SPF50 200ml 22.99, Plasters 40pk 5.99. Add 15% GST (rounded to 2 decimals) to make a new Series, then show only the products whose GST-inclusive price is over $10.

Show a solution ```python import pandas as pd prices = pd.Series( [4.50, 18.99, 22.99, 5.99], index=["Paracetamol 500mg 20pk", "Vitamin C 1000mg 60tab", "Sunscreen SPF50 200ml", "Plasters 40pk"], ) prices_incl = (prices * 1.15).round(2) prices_incl[prices_incl > 10] ``` **Output:** ```text Vitamin C 1000mg 60tab 21.84 Sunscreen SPF50 200ml 26.44 dtype: float64 ```

Core (20 XP each)

Exercise 2.4: Your own DataFrame. Build a DataFrame from this dictionary of lists, add a revenue column, keep only the Cold & Flu rows, and sort them by revenue from highest to lowest.

data = {
    "product": ["Throat Lozenges 16pk", "Paracetamol 500mg 20pk", "Vapour Rub 50g",
                "Cold & Flu Capsules 24pk", "Vitamin C 1000mg 60tab", "Throat Lozenges 16pk"],
    "category": ["Cold & Flu", "Pain Relief", "Cold & Flu", "Cold & Flu", "Vitamins", "Cold & Flu"],
    "unit_price": [8.99, 4.50, 9.99, 14.50, 18.99, 8.99],
    "quantity": [2, 3, 4, 1, 2, 5],
}
Show a solution ```python products = pd.DataFrame(data) products["revenue"] = products["unit_price"] * products["quantity"] cold_flu = products[products["category"] == "Cold & Flu"] cold_flu.sort_values("revenue", ascending=False) ``` **Output:** ```text product category unit_price quantity revenue 5 Throat Lozenges 16pk Cold & Flu 8.99 5 44.95 2 Vapour Rub 50g Cold & Flu 9.99 4 39.96 0 Throat Lozenges 16pk Cold & Flu 8.99 2 17.98 3 Cold & Flu Capsules 24pk Cold & Flu 14.50 1 14.50 ```

Exercise 2.5: Group and total. Using products from Exercise 2.4, show the total quantity sold per category, largest first. Then show the total revenue per category.

Show a solution ```python products.groupby("category")["quantity"].sum().sort_values(ascending=False) ``` **Output:** ```text category Cold & Flu 12 Pain Relief 3 Vitamins 2 Name: quantity, dtype: int64 ``` ```python products.groupby("category")["revenue"].sum().round(2) ``` **Output:** ```text category Cold & Flu 117.39 Pain Relief 13.50 Vitamins 37.98 Name: revenue, dtype: float64 ```

Exercise 2.6: Chart it. Draw a horizontal bar chart of revenue per category (from Exercise 2.5), with a title and axis label, and save it as output/ch02_ex6.png.

Show a solution ```python import matplotlib.pyplot as plt revenue_by_category = products.groupby("category")["revenue"].sum().sort_values() fig, ax = plt.subplots(figsize=(7, 3.5)) revenue_by_category.plot(kind="barh", ax=ax, color="steelblue") ax.set_title("Revenue by category") ax.set_xlabel("Revenue (NZD)") ax.set_ylabel("") fig.savefig("../output/ch02_ex6.png", dpi=150, bbox_inches="tight") plt.show() ``` `ax.set_ylabel("")` removes the automatic "category" label, which adds nothing when the bar names are already there.

Stretch (40 XP each)

Exercise 2.7: Two dimensions. Using branch_week from section 2.4 (4 branches × 7 days), and without any loops, find: (a) each branch's weekly total, (b) each day's total across all branches, (c) the position of each branch's best day, and (d) that best day's name. Hint: argmax also accepts axis=, and a NumPy array of day names can be indexed with an array of positions.

Show a solution ```python branch_week = np.array([ [520, 480, 610, 590, 700, 820, 400], [410, 395, 450, 470, 600, 760, 380], [300, 310, 290, 350, 420, 510, 330], [220, 230, 260, 240, 300, 380, 250], ]) day_names = np.array(["Mon", "Tue", "Wed", "Thu", "Fri", "Sat", "Sun"]) print(branch_week.sum(axis=1)) # (a) per branch print(branch_week.sum(axis=0)) # (b) per day best_positions = branch_week.argmax(axis=1) print(best_positions) # (c) print(day_names[best_positions]) # (d) ``` **Output:** ```text [4120 3465 2510 1880] [1450 1415 1610 1650 2020 2470 1360] [5 5 5 5] ['Sat' 'Sat' 'Sat' 'Sat'] ``` Every branch peaks on Saturday. Indexing `day_names` with an *array* of positions (`day_names[[5, 5, 5, 5]]`) returns the name at each position; this is called **fancy indexing**.

Exercise 2.8: Fairer targets, and a rebuild drill.

  1. In your milestone notebook, replace the single $1,000 target with a target for each branch equal to 110% of its own daily average, and count how many days each branch hit its own target. (Hint: daily.mean() * 1.1 gives one target per branch, and comparing a DataFrame to a Series lines them up by column name.)
  2. Then prove your environment is reproducible: close VS Code/JupyterLab, delete the .venv folder, recreate it (section 1.2.4), reinstall with python -m pip install -r requirements.txt, reopen the milestone notebook, select the new .venv kernel and Restart and Run All. Your numbers should be identical.
Show a solution ```python branch_targets = (daily.mean() * 1.1).round(2) print(branch_targets) days_on_own_target = (daily >= branch_targets).sum() days_on_own_target ``` **Output:** ```text Lambton Quay 1488.30 Cuba Street 1255.26 Petone 986.35 Karori 764.25 dtype: float64 Lambton Quay 6 Cuba Street 7 Petone 8 Karori 6 dtype: int64 ``` When you compare a DataFrame with a Series, pandas matches the Series' labels to the DataFrame's **column names**, so each branch is compared with its own target. This automatic lining-up by label is called **alignment**, and it's one of pandas' quiet superpowers. Now every branch has a comparable measure of "a strong day". For part 2, identical numbers after a full rebuild are the proof: your seed fixed the random data, and `requirements.txt` fixed the library versions.

Quick self-check (20 XP for all five)

  1. What's the difference between [1, 2, 3] * 2 and np.array([1, 2, 3]) * 2?
  2. Why is python -m pip install safer than pip install?
  3. In df[df["quantity"] >= 3], what does the inner part produce, and what does the outer part do with it?
  4. On a 2-D array, which axis gives you one result per row?
  5. Why does df[df["loyalty_member"]]["discount"] = 0.05 fail to change df in pandas 3, and what should you write instead?
Show answers 1. The list repeats itself: `[1, 2, 3, 1, 2, 3]`. The array doubles each element: `[2, 4, 6]` (vectorisation). 2. It guarantees you use the pip belonging to the `python` your terminal finds first, which is your active environment's. Plain `pip` can belong to a different Python on machines with several installed. 3. The inner part produces a boolean mask (one True/False per row). The outer brackets keep only the rows marked True. 4. `axis=1` collapses across each row, giving one result per row. 5. It's chained assignment: the first brackets produce a temporary selection, and in pandas 3 (Copy-on-Write) changing that temporary result never flows back to `df`. Write `df.loc[df["loyalty_member"], "discount"] = 0.05`.

2.11 Cheat sheet

Task Code
Install into the active environment python -m pip install pandas
Record the environment python -m pip freeze > requirements.txt
Rebuild the environment python -m pip install -r requirements.txt
Start JupyterLab jupyter lab
Standard imports import numpy as np, import pandas as pd, import matplotlib.pyplot as plt, import seaborn as sns
Make an array np.array([...]), np.arange(a, b), np.zeros(n)
Array facts .shape, .dtype, .ndim, .size
Summaries .sum(), .mean(), .max(), .argmax(), .std()
Boolean mask arr[arr > 10], (arr > 10).sum(), (a > 1) & (b < 5)
If/else on a whole array np.where(cond, "yes", "no")
Along an axis axis=0 collapses down columns; axis=1 collapses across rows
Reproducible randomness rng = np.random.default_rng(seed=42)
Make a Series / DataFrame pd.Series(values, index=labels), pd.DataFrame(list_of_dicts)
First look df.shape, df.head(), df.dtypes, df.info(), df.describe()
Select columns df["col"] (Series), df[["a", "b"]] (DataFrame)
New column df["revenue"] = df["price"] * df["qty"]
Filter rows df[df["qty"] >= 3]
Sort / count df.sort_values("col", ascending=False), df["col"].value_counts()
Group and total df.groupby("branch")["revenue"].sum()
Change values safely df.loc[condition, "col"] = value
Missing values s.isna().sum()
Chart skeleton fig, ax = plt.subplots() → ax.plot/bar/hist(...) → ax.set_title(...) → plt.show()
Save a chart fig.savefig("../output/name.png", dpi=150, bbox_inches="tight")
Seaborn totals bar chart sns.barplot(data=df, x=..., y=..., estimator="sum", errorbar=None, ax=ax)

2.12 XP tally and what's next

Section XP available XP earned
Warm-up (2.1–2.3) 30
Core (2.4–2.6) 60
Stretch (2.7–2.8) 80
Milestone notebook runs cleanly with "Restart and Run All" 60
Self-check (all five correct) 20
Chapter 2 total 250
Running total (Chapters 1–2) 440

Coming up in Chapter 3: Loading, Inspecting and Selecting Data. The real Harbour Pharmacy sales file arrives. You'll learn how file paths work, load CSV, Excel and JSON files with pandas, run a systematic first-look checklist that uncovers the problems hiding in real data, and master .loc, .iloc and boolean filtering to pull out exactly the rows and columns a question needs.


References and further reading