Session 3

  • map and filter
  • Modules and packages
  • Files
  • Exceptions
  • Classes
  • Debugging

Learning outcomes

Learning outcomes: map and filter

You’ll be able to:

  • Use map to apply functions to iterables (e.g. list, dict)
  • Filter data with filter and pandas masks

Learning outcomes: Modules and packages

You’ll be able to:

  • Create, import and use your own modules

Learning outcomes: Files

You’ll be able to:

  • Open, read and write to files
  • Use context managers with files
  • Manipulate binary files with pickle
  • Read and write in JSON format
  • Use the pathlib library to simplify file operations

Learning outcomes: Exceptions

You’ll be able to:

  • read tracebacks to find and fix errors
  • catch exceptions to prevent program crashes

Learning outcomes: Classes

You’ll be able to:

  • understand the basic concepts of Object Oriented Programming
  • create basic classes
  • use classes from other packages

Learning outcomes: Debugging

You’ll be able to:

  • use the debugger to find and fix errors

Unit 8

map and filter

map and filter

  • map, filter
  • filters in pandas: masks

map

Let’s suppose we have a list of prices:

prices = [1.92, 46.4, 30.56, 11.63, 8.35,
          6.32, 28.21, 8.21, 12.86, 6.95]

These prices will suffer an update of 4.5%. How can we update them?

With what we’ve learned so far, we could write a loop.

new_prices = []
percent = 4.5
for p in prices:
    new_prices.append(p * (1 + percent/100))

Or, we can use map, which applies a function to all elements of an iterable (e.g. list).

percent = 4.5
def update(price):
    return price * (1 + percent/100)

new_prices = list(map(update, prices))

Or use lambda…

percent = 4.5
new_prices = list(map(lambda p: p * (1 + percent / 100), prices))

⚠️ Keep your lambdas short. ⚠️

If you have complex logic, transfer it to a “normal” function where the logic will be clearer.

Why are we converting the result of map to a list?

map actually returns an iterable, not a list (like zip and enumerate)

In [11]: map(lambda p: p * (1 + percent / 100), prices)
Out[11]: <map at 0x15a9b31cb80>

And it can also receive any iterable (tuple, dict, …), not just a list.

Another example:

authors = dict(zip(names, ages))
output = list(map(lambda n, a: f"Author {n} died at age {a}.",
                  authors.items()))
# equivalent to 
output = [f"Author {n} died at age {a}." for n,a in authors.items()]

Are you really going to use this?

Often people don’t directly use Python’s default map, but map is often implemented in other data structures, such as…

df = pandas.read_excel("prices.xlsx")
In [45]: df.head()
Out[45]: 
   prices    category
0   22.25  vegetables
1   10.22  vegetables
2    2.19  vegetables
3   14.63        pets
4   31.21       fruit
In [46]: df.shape
Out[46]: (50, 2)
df["new_prices"] = df["prices"].map(lambda p: p * 1.045)
In [51]: df.head()
Out[51]: 
   prices    category  new_prices
0   22.25  vegetables    23.25125
1   10.22  vegetables    10.67990
2    2.19  vegetables     2.28855
3   14.63        pets    15.28835
4   31.21       fruit    32.61445
In [52]: df.shape
Out[52]: (50, 3)

filter

filter works just like map

but instead of applying a function to every value, we’re picking which values we want to keep.

Because we’re choosing which values to keep, the function that we give to filter must return a bool (True or False).

prices = [9.56, 23.82, 17.39, 5.11, 8.28]
categories = ['fruit', 'hygiene', 'fruit', 'vegetables', 'vegetables']

vegetables_prices = [p for p,c in zip(prices, categories) \
                     if c == 'vegetables']
vegetables_prices = list(filter(lambda p,c: c == 'vegetables',
                                zip(prices, categories)))

Wil you use this?

Probably not! In Pandas, we actually filter in different ways, like:

df = pandas.read_excel("prices.xlsx")
vegetables = df[df["category"] == "vegetable"]
In [61]: vegetables.head()
Out[61]: 
    prices    category  new_prices
0    22.25  vegetables    23.25125
1    10.22  vegetables    10.67990
2     2.19  vegetables     2.28855
13    1.59  vegetables     1.66155
22   14.17  vegetables    14.80765
In [62]: vegetables.shape
Out[62]: (13, 3)
In [46]: df.shape
Out[46]: (50, 2)

Why does that work?

When we compare the column “category” with the value “vegetables”, we get another column of the same size indicating where that comparison was True or False.

In [64]: df["category"] == "vegetables"
Out[64]: 
0      True
1      True
2      True
3     False
4     False
5     False

This column is the mask that will select the rows of the original dataframe where the comparison is True.

Unit 9

Modules and packages

Modules and packages

  • modules & packages
  • import our own code

External code

Python modules

In Python, we can create our own modules to store code and import that code into other programs.

Why do this? Reusability and modularity.

You can download the notebook here and read it directly in Github.

Unit 10

Files

Files

  • files (open, close, with, read, write)
  • binary files: pickle
  • JSON: json.dump, json.load, json.dumps, json.loads
  • pathlib: swiss army knife for files

Files

text

  • open, read, write, close
  • modes r, w, a
  • context manager
  • pathlib, swiss army knife for files

open

f = open("test_file.txt")  # open a file

This gives an error, because this file does not exist.

In [66]: f = open("test_file.txt")  # open a file
---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
<ipython-input-66-a7d550966829> in <module>
----> 1 f = open("test_file.txt")  # open a file

FileNotFoundError: [Errno 2] No such file or directory: 'test_file.txt'

There are 3 ways of opening a file:

  • “r” for reading only, the file must exist exist
  • “w” for writing, creates the file if it doesn’t exist or overwrites, starts writing at the beginning
  • “a” same as “w”, but starts writing at the end

If a mode is not speficied when opening a file, the default is “r”. That is why we got an error.

Let’s create our file first.

f = open("test_file.txt", "w")  # open a file in write mode
f.write("line 1\n") # \n write a newline sequence to the file
f.write("line 2\n")
f.close()  # we must close the file for the changes to take effect

Now, let’s read it.

f = open("test_file.txt", "r")  # open a file in write mode
data = f.read()
print(data)
f.close()
line 1
line 2

We can add a + modifier to any mode, and it allows the modes to both read and write.

"r+" still doesn’t create a file if it doesn’t exist though.

It’s easier to forget closing files, specially when there are many operations between opening and closing.

The recommended way of opening and closing files is to use a context manager

with open("test_file.txt", "r") as f:
    data = f.read()
print(data)

When we use with, the file is automatically closed when the block of instructions inside with is finished.

Binary files

Binary files store data in a format that is not human readable.

They’re very useful to store numeric data because we can fit more numbers within the same memory space.

To use files in binary mode, we simply add a “b” to the modes we’ve already seen so far.

with open("test.pkl", "rb") as f:
    # do something...

They will also be useful to store trained models to be deployed or used by other people.

In Python, we can store almost any object in a binary format with the pickle module.

Let’s create a dict to store in a binary file.

test_dict = {x: y for x,y in zip(range(0,5), range(100,105))}
{0: 100,
 1: 101,
 2: 102,
 3: 103,
 4: 104}

Storing an object in a pickle file

import pickle
with open("test.pkl", "wb") as f:
    pickle.dump(test_dict, f)

Loading an object from a pickle file

with open("test.pkl", "rb") as f:
    loaded_dict = pickle.load(f)
print(loaded_dict)
{0: 100,
 1: 101,
 2: 102,
 3: 103,
 4: 104}

JSON

JSON (JavaScript Object Notation) is a human-readable text format for storing structured data.

It maps naturally to Python dicts and lists.

import json

data = {"name": "Ana", "scores": [90, 85, 92]}

# Write to file
with open("data.json", "w") as f:
    json.dump(data, f)

# Read from file
with open("data.json", "r") as f:
    loaded = json.load(f)

print(loaded)  # -> {'name': 'Ana', 'scores': [90, 85, 92]}

We can also convert to and from JSON strings without touching a file.

import json

data = {"name": "Ana", "scores": [90, 85, 92]}

s = json.dumps(data)   # dict -> JSON string
# -> '{"name": "Ana", "scores": [90, 85, 92]}'

parsed = json.loads(s) # JSON string -> dict

JSON vs pickle

JSON pickle
Human-readable ✅ ❌
Cross-language ✅ ❌
Any Python object ❌ ✅

Use JSON for configs, APIs, and data exchange (not storage!).

Use pickle for Python-specific objects (e.g. trained models).

pathlib

I recommend the use of the pathlib library anytime you need to deal with files.

It makes working with files much easier, since there are things that change when our code executes in Windows, MacOS, Linux, etc.

By using pathlib, many of these differences are abstracted away.

We also have many utilities that make life easier.

create a Path

Path is the main object to use, and it provides many methods to streamline file operations.

from pathlib import Path
test_path = Path("test_file.txt")
test_path.exists() # returns True if the file exists

open, write, read

test_path = Path("test_file.txt")
with test_path.open("a") as f:
    f.write("line 3\n")

with test_path.open("a") as f:
    data = f.readlines()
print(data)

Output:

['line 1\n', 'line 2\n', 'line 3\n']

.readlines() is a new method, that reads the whole file and splits the content by lines in a list.

read and write “shortcuts”

print(test_path.read_text())

# we can't pass a mode to .write_text
test_path.write_text("overwrites everything")

print(test_path.read_text())

Output of line 1:

line 1
line 2
line 3

Output of line 6:

overwrites everything

They also work with paths that are directories and it becomes easier to list all files inside.

p = Path("code")
p.is_dir()
True

List all files and folders inside code/

list(p.iterdir())
[WindowsPath('code/.ipynb_checkpoints'),
 WindowsPath('code/conditionals1.py'),
 WindowsPath('code/conditionals2.py'),
 WindowsPath('code/io_terminal.py'),
 WindowsPath('code/lesson3'),
 WindowsPath('code/types.py'),
 WindowsPath('code/variables.py')]

List all .py files inside code/, recursively.

list(p.rglob("*.py")) # rglob = recursive glob
[WindowsPath('code/conditionals1.py'),
 WindowsPath('code/conditionals2.py'),
 WindowsPath('code/io_terminal.py'),
 WindowsPath('code/types.py'),
 WindowsPath('code/variables.py')]

Path objects have many more useful methods, but we won’t explore them in this course:

  • creating and deleting files/folders
  • retrieving parent folders
  • retrieving absolute and relative paths
  • concatenating paths
  • …

Unit 11

Exceptions

Exceptions

  • catching errors
  • reading tracebacks

Exceptions

  • Unexpected behaviour can manifest itself as an exception.
  • If not dealt with, exceptions break our programs.
int("hello world")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: invalid literal for int() with base 10: 'hello'
  • Converting a nonnumeric string to an integer is unexpected behaviour, and a ValueError exception is thrown.
read_my_file("imaginary_file.txt")
FileNotFoundError: [Errno 2] No such file or directory: 'imaginary_file.txt'
  • The file imaginary_file.txt does not exist, so a FileNotFoundError exception is thrown.
  • Python has many different exceptions.
  • Their name is usually sugestive of the problem we’re facing.
  • Many packages define their own exceptions.
  • Sometimes those are not as suggestive.

When we get an exception, we get:

  1. the type of the exception, e.g. FileNotFoundError
  2. usually, a suggestive string, e.g.  No such file or directory: 'imaginary_file.txt'
  3. a traceback

traceback

  • The traceback is the trace of instructions from our program’s origin to the point where the exception happened.
  • If an exception was thrown inside a function:
    • the traceback includes the line where the function was called
    • and then the line inside the function where the exception was thrown.
    • recursively

There may be a long list of function calls.

Traceback (most recent call last):
  File "script.py", line 8, in <module>
    avs = averages(data_fn)
  File "mypackage/stats.py", line 17, in averages
    data = read_csv(filename)
  File "mypackage/file_utils.py", line 7, in read_csv
    with open(fn) as f:
FileNotFoundError: [Errno 2] No such file or directory: 'imaginary_file.txt'

traceback

It’s a traceback because it allows us to trace the problem back to its root.

Catching exceptions

  • When an exception happens, we can have a backup plan.
  • In Python, we can use a try block to catch an exception and stop the program from failing.
try:
    number = int("hello")
except:
    print("There was some error, but I'm not leaving")
    number = 42

We can catch just some kinds of exceptions too.

try:
    number = int("hello")
except ValueError:
    print("we got an invalid number, proceeding with default 42")
    number = 42

We can also catch more exceptions or combine them.

try:
    number = int("hello")
except (ValueError, TypeError):
    print("there's something wrong with the value or the type. default 42")
    number = 42
except NotImplementedError:
    print("This feature is not implemented yet, sorry.")
except:
    print("I'm catching every other kind of exception :)")

Unit 12

Classes

Classes

  • classes and objects
  • attributes and methods
  • inheritance

Classes

  • We’ve already been exposed to objects, but where do they come from?

Object Oriented Programming (OOP)

  • Python allows programming in the Object Oriented paradigm.
  • This means we can define classes of objects that share attributes and behaviour
  • Behaviour is implemented through methods (functions associated with a class) 
class Vehicle:
    def __init__(self, color): # special method, the constructor
        self.color = color  # this is an attribute
        self.acc = 0        # another attribute

    def accelerate(self):   # this is a method
        self.acc += 2       # fetching a predefined
                            # attribute and modifying it

    def deaccelerate(self): # another method
        self.acc -= 2

instatiating classes

To use a class, we call it like we call functions.

v = Vehicle("red")
print(v.color)
v.accelerate()
print(v.acc) # -> 2
v.deaccelerate()
print(type(v))
red
2
<class '__main__.Vehicle'>
  • When we create an object of a certain class, we call it instantiation.
  • Classes by themselves don’t do anything, but their instantiated objects do.
    • The concept of a Car doesn’t take us anywhere, but a specific (instantiated) car can.

classes from other packages

Remember the Pandas DataFrame?

df = pandas.read_excel("prices.xlsx")
type(df)
pandas.core.frame.DataFrame

It’s a class!

  • The read_excel function reads the contents of the Excel file into a newly created DataFrame object.
  • When we call the methods of an object, we’re calling the methods of that object’s class!
df["new_prices"] = df["prices"].map(lambda p: p * 1.045)
df["new_prices"] = df["prices"].map(lambda p: p * 1.045)
  • We have 3 methods here:
    • map is a method.
    • getting a column with [ ], is the implementation of the special method __getitem__
    • setting a column with [ ] is also the implementation of __setitem__.

inheritance

  • It’s also possible for classes to inherit attributes and behaviour from other classes.
  • They can modify only part of the behaviour of the parent classes.
class Motorcycle(Vehicle):
    def accelerate(self):
        self.acc += 10

m = Motorcycle("red")
print(m.color)
m.accelerate()
print(m.acc)
red
10
  • You’ll implicintly use this when using Python’s packages.
    • In SciKit-Learn, the LinearRegression class inherits from MultiOutputMixin, RegressorMixin and LinearModel.
    • In the same package, the DecisionTreeClassifier inherits from ClassifierMixin and BaseDecisionTree.
  • And also explicitly:
    • In PyTorch, it’s common to build models by creating classes that inherit from torch.nn.Module.

What you need to know about OOP?

  • Classes are recipes for the creation of objects: they specify structure and behaviour.
  • Data libraries will often return objects, e.g. Pandas DataFrame
  • The classes of these objects usually have useful methods, e.g. .map() of DataFrame
  • Sometimes libraries expect values that can have the type of any child from a particular class (inheritance).
sklearn.linear_model.LinearRegression.__bases__
(sklearn.base.MultiOutputMixin,
 sklearn.base.RegressorMixin,
 sklearn.linear_model._base.LinearModel)
  • The LinearRegression class from SciKit-Learn inherits from 3 classes.
  • Some SciKit-Learn functions can receive any object from a class that inherits from RegressorMixin.

Unit 13

Debugging

Debugging

  • pdb
  • debugging in Jupyter and IDEs

debugging

  • In code: import pdb; pdb.set_trace() or breakpoint()
  • python -m pdb some_program.py
  • Built-in debuggers
  • Jupyter

pdb cheatsheet

import pdb
pdb.set_trace()
python3 -m pdb my_script.py
(c)ontinue - continues the execution
(n)ext     - execute the next line
(s)tep     - step into a function
(w)here    - print traceback and show current frame
(u)p       - go up one frame in traceback
(d)own     - go down one frame in traceback
(p)rint    - evaluate expression
(b)reak    - set breakpoint (write line)

Recap

Recap: map and filter

map

apply a function to a “list” of values

percent = 4.5
new_prices = list(map(lambda p: p * (1 + percent / 100), prices))

pandas and map

df = pandas.read_excel("prices.xlsx")
df["new_prices"] = df["prices"].map(lambda p: p * 1.045)

Recap: Modules and packages

modules and packages

  • Python (.py) files in the same folder as our program can be directly imported.
  • We can organize code in modules, and modules in packages (don’t forget those __init__.py files)
retailxy/
 |- __init__.py
 |- products.py
 |
 |- marketing/
 |    |- __init__.py
 |    |- discounts.py
 |    |- promotions.py
import retailxy.marketing as mark
from products import *

Recap: Files

files

f = open("test_file.txt", "w+")  # open a file
data = f.read()
f.write("\njust one more line")
f.close() # don't forget to close your files

context manager

with open("test_file.txt", "r") as f:
    data = f.read()
print(data)

The file is closed for us at the end of the context (with block)

binary

We can store (almost) any Python object in a binary file with pickle

import pickle

with open("test.pkl", "wb") as f:
    pickle.dump(test_dict, f)

with open("test.pkl", "rb") as f:
    loaded_dict = pickle.load(f)
print(loaded_dict)

JSON

Human-readable, cross-language format. Maps to dicts and lists.

import json

# file
with open("data.json", "w") as f:
    json.dump(data, f)      # write dict to file
with open("data.json", "r") as f:
    loaded = json.load(f)   # read dict from file

# string
s = json.dumps(data)        # dict -> JSON string
parsed = json.loads(s)      # JSON string -> dict

pathlib - Path

from pathlib import Path
test_path = Path("test_file.txt")
test_path.exists() # returns True if the file exists

pathlib - open, read, write

test_path = Path("test_file.txt")
with test_path.open("a") as f:
    f.write("line 3\n")

with test_path.open("a") as f:
    data = f.readlines()
print(data)

pathlib - many utilities

p.is_dir()  # is this path a directory?
list(p.iterdir())  # lists all files in directory
list(p.glob("*.py"))  # lists all .py files in the directory

Recap: Exceptions

Exceptions

int("hello")  # -> ValueError

open("missing_file.txt")  # -> FileNotFoundError

my_list[100]  # -> IndexError

When something unexpected happens, Python throws an exception.

If not handled, exceptions stop program execution.

Reading Tracebacks

Traceback (most recent call last):
  File "script.py", line 8, in <module>
    result = process_data(filename)
  File "utils.py", line 15, in process_data
    data = load_file(fn)
  File "utils.py", line 5, in load_file
    with open(fn) as f:
FileNotFoundError: No such file or directory: 'data.txt'

Read from bottom to top: - What exception? FileNotFoundError - Where did it happen? utils.py, line 5, inside load_file - How did we get there? Called from line 15, called from line 8

Catching Exceptions

try:
    number = int(user_input)
except ValueError:
    print("Invalid number, using default")
    number = 0
try:
    data = load_file(filename)
except FileNotFoundError:
    print(f"File {filename} not found")
    data = []
except PermissionError:
    print("No permission to read file")
    data = []

Multiple Exception Handling

try:
    result = risky_operation()
except (ValueError, TypeError):
    print("Problem with value or type")
except FileNotFoundError:
    print("File missing")
except:
    print("Some other error occurred")

⚠️ Catch specific exceptions when possible!

Recap: Classes

Classes - Basic Structure

class Model:
    def __init__(self, name, learning_rate):  # constructor
        self.name = name              # attribute
        self.learning_rate = learning_rate
        self.trained = False          # attribute
    
    def train(self, data):           # method
        # training logic here
        self.trained = True
    
    def predict(self, features):     # method
        if not self.trained:
            raise ValueError("Model not trained yet")
        # prediction logic here

Using Classes

# Instantiate (create an object)
my_model = Model("LinearRegression", 0.01)

# Access attributes
print(my_model.name)            # -> "LinearRegression"
print(my_model.learning_rate)   # -> 0.01

# Call methods
my_model.train(training_data)
predictions = my_model.predict(test_features)

Classes We Already Use

import pandas as pd

df = pd.DataFrame(data)  # DataFrame is a class!
type(df)  # -> pandas.core.frame.DataFrame

df.head()        # method
df.describe()    # method
df["column"]     # special method __getitem__

Everything in Python is an object from some class!

Inheritance

class BaseModel:
    def __init__(self, name):
        self.name = name
        self.trained = False
    
    def train(self, data):
        self.trained = True

class NeuralNetwork(BaseModel):  # inherits from BaseModel
    def train(self, data):       # override parent method
        print("Training neural network...")
        super().train(data)      # call parent's train method

Inheritance in Practice

Many ML libraries use inheritance:

# PyTorch example structure
class MyModel(torch.nn.Module):  # inherit from Module
    def __init__(self):
        super().__init__()
        # define layers
    
    def forward(self, x):
        # define forward pass
        return output
# Scikit-learn classifiers inherit from BaseEstimator
LinearRegression inherits from: RegressorMixin, LinearModel
DecisionTreeClassifier inherits from: ClassifierMixin, BaseDecisionTree

Recap: Debugging

Debugging with pdb

import pdb

def calculate_average(numbers):
    pdb.set_trace()  # execution pauses here
    total = sum(numbers)
    avg = total / len(numbers)
    return avg

Or run script in debug mode:

python -m pdb my_script.py

pdb Commands

(n)ext      - execute next line
(s)tep      - step into function
(c)ontinue  - continue execution
(p)rint var - print variable value
(l)ist      - show current code
(w)here     - show traceback
(b)reak 10  - set breakpoint at line 10
(q)uit      - quit debugger

Use pdb to inspect variables and understand program flow!

Debugging in Jupyter and IDE

  • IDEs usually have debugging modes
  • Set breakpoints more easily
  • Facilitate moving up and down the call stack
  • Easily check variable values
// multiplex-patch.js // Patches the reveal.js multiplex plugin to also broadcast: // - fragment navigation (fragmentshown / fragmenthidden) // - slide changes for headingless slides (re-emits slidechanged robustly) // // Only activates on the master (speaker view) — clients are unaffected. // Include in your Quarto frontmatter under revealjs.include-after-body // or revealjs.include-in-header (after the multiplex plugin loads). Reveal.on('ready', () => { const config = Reveal.getConfig().multiplex; // Only patch the master (speaker view has the secret) if (!config?.secret) return; const socket = io.connect(config.url); const broadcast = () => { socket.emit('multiplex-statechanged', { state: Reveal.getState(), // includes indexh, indexv, indexf, paused, overview secret: config.secret, socketId: config.id, }); }; Reveal.on('fragmentshown', broadcast); Reveal.on('fragmenthidden', broadcast); // Re-emit on slidechanged to catch headingless slides that the // stock plugin may miss due to index resolution timing Reveal.on('slidechanged', broadcast); });