Get In Touch
0% complete

Data analysis and machine learning with Python · Lesson 1 of 3

Numerical computing with NumPy and pandas

Represent arrays and tabular data while checking types, missing values, and dimensions.

Lesson

1. Concept and mental model

NumPy provides multidimensional arrays and vectorized operations for numerical computing. A NumPy array has a shape and data type; inspect both when a calculation produces an unexpected result. Vectorized expressions operate on whole arrays and are often clearer and faster than Python loops over every number. Broadcasting allows compatible shapes to work together, but verify the resulting dimensions so a calculation does not silently combine the wrong axes.

2. How it works

pandas provides labeled Series and DataFrame structures for tabular data. Read a file with an explicit schema where possible, inspect columns and missing values, and avoid assuming every column was inferred correctly. Cleaning may include parsing dates, normalizing categories, handling missing values, and removing duplicates according to a domain rule. Keep the original source or a reproducible ingestion step so cleaning decisions can be audited.

3. Use it safely and effectively

Do not treat a successful transformation as proof that data is correct. Check row counts, ranges, unique identifiers, and sample records before and after important operations. For data larger than memory, consider chunked processing or a distributed tool rather than repeatedly copying full DataFrames. Practice by reading a small sales table, grouping by category, and verifying the aggregate against a hand-calculated example.

4. Syntax reference

import pandas as pd

data = pd.DataFrame({'category': ['book', 'pen', 'book'], 'amount': [20, 5, 10]})
print(data.groupby('category')['amount'].sum().to_dict())

5. Guided walkthrough

Load a representative sample, inspect its shape and data types, and identify missing or malformed values before calculating. Use vectorized NumPy or pandas operations for column-wise work, and verify the result against a small hand-calculated example.

6. Common mistakes to avoid

  • Assuming imported columns have the expected numeric type without inspecting them.
  • Using a Python loop for a large column operation when a vectorized expression is clearer and faster.

Visual guide

Prepare a dataset

  1. Step 01 Load

    Read data and inspect its schema.

  2. Step 02 Clean

    Apply documented missing-value and type rules.

  3. Step 03 Verify

    Check dimensions and representative aggregates.

Code examples

Worked example
PYTHON 
import pandas as pd

data = pd.DataFrame({'category': ['book', 'pen', 'book'], 'amount': [20, 5, 10]})
print(data.groupby('category')['amount'].sum().to_dict())
Expected output
{'book': 30, 'pen': 5}
Calculate array values with NumPy
PYTHON 
import numpy as np

scores = np.array([70, 80, 90])
print(scores.mean())
Expected output
80.0
Summarize a pandas column
PYTHON 
import pandas as pd

df = pd.DataFrame({'team': ['A', 'A', 'B'], 'points': [2, 3, 4]})
print(df.groupby('team')['points'].sum().to_dict())
Expected output
{'A': 5, 'B': 4}

Key points

  • Represent arrays and tabular data while checking types, missing values, and dimensions.
  • Prefer explicit, testable behavior over hidden side effects.
  • Validate assumptions at system boundaries and handle failure deliberately.

Check your understanding

Practice

What should you inspect before trusting a DataFrame calculation?

Your learning progress

Finish this lesson