zev/practical-python
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

link experiment

Browse Source
This commit is contained in:
David Beazley
2020-05-26 09:21:19 -05:00
parent 9aec32e1a9
commit b5244b0e61
24 changed files with 4265 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

9
Notes/07_Advanced_Topics/00_Overview.md Normal file
View File

@@ -0,0 +1,9 @@
# Overview
In this section we will take a look at more Python features you may encounter.
* Variable argument functions
* Anonymous functions and lambda
* Returning function and closures
* Function decorators
* Static and class methods

214
Notes/07_Advanced_Topics/01_Variable_arguments.md Normal file
View File

@@ -0,0 +1,214 @@
# 7.1 Variable Arguments
### Positional variable arguments (*args)
A function that accepts *any number* of arguments is said to use variable arguments.
For example:
```python
def foo(x, *args):
...
```
Function call.
```python
foo(1,2,3,4,5)
```
The arguments get passed as a tuple.
```python
def foo(x, *args):
# x -> 1
# args -> (2,3,4,5)
```
### Keyword variable arguments (**kwargs)
A function can also accept any number of keyword arguments.
For example:
```python
def foo(x, y, **kwargs):
...
```
Function call.
```python
foo(2,3,flag=True,mode='fast',header='debug')
```
The extra keywords are passed in a dictionary.
```python
def foo(x, y, **kwargs):
# x -> 2
# y -> 3
# kwargs -> { 'flat': True, 'mode': 'fast', 'header': 'debug' }
```
### Combining both
A function can also combine any number of variable keyword and non-keyword arguments.
Function definition.
```python
def foo(*args, **kwargs):
...
```
This function takes any combination of positional or keyword arguments.
It is sometimes used when writing wrappers or when you want to pass arguments through to another function.
### Passing Tuples and Dicts
Tuples can be expanded into variable arguments.
```python
numbers = (2,3,4)
foo(1, *numbers) # Same as f(1,2,3,4)
```
Dictionaries can also be expaded into keyword arguments.
```python
options = {
'color' : 'red',
'delimiter' : ',',
'width' : 400
}
foo(data, **options)
# Same as foo(data, color='red', delimiter=',', width=400)
```
These are not commonly used except when writing library functions.
## Exercises
### (a) A simple example of variable arguments
Try defining the following function:
```python
>>> def avg(x,*more):
return float(x+sum(more))/(1+len(more))
>>> avg(10,11)
10.5
>>> avg(3,4,5)
4.0
>>> avg(1,2,3,4,5,6)
3.5
>>>
```
Notice how the parameter `*more` collects all of the extra arguments.
### (b) Passing tuple and dicts as arguments
Suppose you read some data from a file and obtained a tuple such as
this:
```
>>> data = ('GOOG', 100, 490.1)
>>>
```
Now, suppose you wanted to create a `Stock` object from this
data. If you try to pass `data` directly, it doesn't work:
```
>>> from stock import Stock
>>> s = Stock(data)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: __init__() takes exactly 4 arguments (2 given)
>>>
```
This is easily fixed using `*data` instead. Try this:
``python
>>> s = Stock(*data)
>>> s
Stock('GOOG', 100, 490.1)
>>>
```
If you have a dictionary, you can use `**` instead. For example:
```python
>>> data = { 'name': 'GOOG', 'shares': 100, 'price': 490.1 }
>>> s = Stock(**data)
Stock('GOOG', 100, 490.1)
>>>
```
### (c) Creating a list of instances
In your `report.py` program, you created a list of instances
using code like this:
```python
def read_portfolio(filename):
'''
Read a stock portfolio file into a list of dictionaries with keys
name, shares, and price.
'''
with open(filename) as lines:
portdicts = fileparse.parse_csv(lines,
select=['name','shares','price'],
types=[str,int,float])
portfolio = [ Stock(d['name'], d['shares'], d['price'])
for d in portdicts ]
return Portfolio(portfolio)
```
You can simplify that code using `Stock(**d)` instead. Make that change.
### (d) Argument pass-through
The `fileparse.parse_csv()` function has some options for changing the
file delimiter and for error reporting. Maybe you'd like to expose those
options to the `read_portfolio()` function above. Make this change:
```
def read_portfolio(filename, **opts):
'''
Read a stock portfolio file into a list of dictionaries with keys
name, shares, and price.
'''
with open(filename) as lines:
portdicts = fileparse.parse_csv(lines,
select=['name','shares','price'],
types=[str,int,float],
**opts)
portfolio = [ Stock(**d) for d in portdicts ]
return Portfolio(portfolio)
```
Once you've made the change, trying reading a file with some errors:
```python
>>> import report
>>> port = report.read_portfolio('Data/missing.csv')
Row 4: Couldn't convert ['MSFT', '', '51.23']
Row 4: Reason invalid literal for int() with base 10: ''
Row 7: Couldn't convert ['IBM', '', '70.44']
Row 7: Reason invalid literal for int() with base 10: ''
>>>
```
Now, try silencing the errors:
```python
>>> import report
>>> port = report.read_portfolio('Data/missing.csv', silence_errors=True)
>>>
```
[Next](02_Anonymous_function)

161
Notes/07_Advanced_Topics/02_Anonymous_function.md Normal file
View File

@@ -0,0 +1,161 @@
# 7.2 Anonymous Functions and Lambda
### List Sorting Revisited
Lists can be sorted *in-place*. Using the `sort` method.
```python
s = [10,1,7,3]
s.sort() # s = [1,3,7,10]
```
You can sort in reverse order.
```python
s = [10,1,7,3]
s.sort(reverse=True) # s = [10,7,3,1]
```
It seems simple enough. However, how do we sort a list of dicts?
```python
[{'name': 'AA', 'price': 32.2, 'shares': 100},
{'name': 'IBM', 'price': 91.1, 'shares': 50},
{'name': 'CAT', 'price': 83.44, 'shares': 150},
{'name': 'MSFT', 'price': 51.23, 'shares': 200},
{'name': 'GE', 'price': 40.37, 'shares': 95},
{'name': 'MSFT', 'price': 65.1, 'shares': 50},
{'name': 'IBM', 'price': 70.44, 'shares': 100}]
```
By what criteria?
You can guide the sorting by using a *key function*. The *key function* is a function that receives the dictionary and returns the value in a specific key.
```python
def stock_name(s):
return s['name']
portfolio.sort(key=stock_name)
```
The value returned by the *key function* determines the sorting.
```python
# Check how the dictionaries are sorted by the `name` key
[
{'name': 'AA', 'price': 32.2, 'shares': 100},
{'name': 'CAT', 'price': 83.44, 'shares': 150},
{'name': 'GE', 'price': 40.37, 'shares': 95},
{'name': 'IBM', 'price': 91.1, 'shares': 50},
{'name': 'IBM', 'price': 70.44, 'shares': 100},
{'name': 'MSFT', 'price': 51.23, 'shares': 200},
{'name': 'MSFT', 'price': 65.1, 'shares': 50}
]
```
### Callback Functions
Callback functions are often short one-line functions that are only used for that one operation. For example of previous sorting example.
Programmers often ask for a short-cut, so is there a shorter way to specify custom processing for `sort()`?
### Lambda: Anonymous Functions
Use a lambda instead of creating the function.
In our previous sorting example.
```python
portfolio.sort(key=lambda s: s['name'])
```
This creates an *unnamed* function that evaluates a *single* expression.
The above code is much shorter than the initial code.
```python
def stock_name(s):
return s['name']
portfolio.sort(key=stock_name)
# vs lambda
portfolio.sort(key=lambda s: s['name'])
```
### Using lambda
* lambda is highly restricted.
* Only a single expression is allowed.
* No statements like `if`, `while`, etc.
* Most common use is with functions like `sort()`.
## Exercises
Read some stock portfolio data and convert it into a list:
```python
>>> import report
>>> portfolio = list(report.read_portfolio('Data/portfolio.csv'))
>>> for s in portfolio:
print(s)
Stock('AA', 100, 32.2)
Stock('IBM', 50, 91.1)
Stock('CAT', 150, 83.44)
Stock('MSFT', 200, 51.23)
Stock('GE', 95, 40.37)
Stock('MSFT', 50, 65.1)
Stock('IBM', 100, 70.44)
>>>
```
### (a) Sorting on a field
Try the following statements which sort the portfolio data
alphabetically by stock name.
```python
>>> def stock_name(s):
return s.name
>>> portfolio.sort(key=stock_name)
>>> for s in portfolio:
print(s)
... inspect the result ...
>>>
```
In this part, the `stock_name()` function extracts the name of a stock from
a single entry in the `portfolio` list. `sort()` uses the result of
this function to do the comparison.
### (b) Sorting on a field with lambda
Try sorting the portfolio according the number of shares using a
`lambda` expression:
```python
>>> portfolio.sort(key=lambda s: s.shares)
>>> for s in portfolio:
print(s)
... inspect the result ...
>>>
```
Try sorting the portfolio according to the price of each stock
```python
>>> portfolio.sort(key=lambda s: s.price)
>>> for s in portfolio:
print(s)
... inspect the result ...
>>>
```
Note: `lambda` is a useful shortcut because it allows you to
define a special processing function directly in the call to `sort()` as
opposed to having to define a separate function first (as in part a).
[Next](03_Returning_functions)

234
Notes/07_Advanced_Topics/03_Returning_functions.md Normal file
View File

@@ -0,0 +1,234 @@
# 7.3 Returning Functions
This section introduces the idea of closures.
### Introduction
Consider the following function.
```python
def add(x, y):
def do_add():
print('Adding', x, y)
return x + y
return do_add
```
This is a function that returns another function.
```python
>>> a = add(3,4)
>>> a
<function do_add at 0x6a670>
>>> a()
Adding 3 4
7
```
### Local Variables
Observe how to inner function refers to variables defined by the outer function.
```python
def add(x, y):
def do_add():
# `x` and `y` are defined above `add(x, y)`
print('Adding', x, y)
return x + y
return do_add
```
Further observe that those variables are somehow kept alive after `add()` has finished.
```python
>>> a = add(3,4)
>>> a
<function do_add at 0x6a670>
>>> a()
Adding 3 4 # Where are these values coming from?
7
```
### Closures
When an inner function is returned as a result, the inner function is known as a *closure*.
```python
def add(x, y):
# `do_add` is a closure
def do_add():
print('Adding', x, y)
return x + y
return do_add
```
*Essential feature: A closure retains the values of all variables needed for the function to run properly later on.*
### Using Closures
Closure are an essential feature of Python. However, their use if often subtle.
Common applications:
* Use in callback functions.
* Delayed evaluation.
* Decorator functions (later).
### Delayed Evaluation
Consider a function like this:
```python
def after(seconds, func):
time.sleep(seconds)
func()
```
Usage example:
```python
def greeting():
print('Hello Guido')
after(30, greeting)
```
`after` executes the supplied function... later.
Closures carry extra information around.
```python
def add(x, y):
def do_add():
print('Adding %s + %s -> %s' % (x, y, x + y))
return do_add
def after(seconds, func):
time.sleep(seconds)
func()
after(30, add(2, 3))
# `do_add` has the references x -> 2 and y -> 3
```
A function can have its own little environment.
### Code Repetition
Closures can also be used as technique for avoiding excessive code repetition.
You can write functions that make code.
## Exercises
### (a) Using Closures to Avoid Repetition
One of the more powerful features of closures is their use in
generating repetitive code. If you refer back to exercise 5.2
recall the code for defining a property with type checking.
```python
class Stock(object):
def __init__(self, name, shares, price):
self.name = name
self.shares = shares
self.price = price
...
@property
def shares(self):
return self._shares
@shares.setter
def shares(self, value):
if not isinstance(value, int):
raise TypeError('Expected int')
self._shares = value
...
```
Instead of repeatedly typing that code over and over again, you can
automatically create it using a closure.
Make a file `typedproperty.py` and put the following code in
it:
```python
# typedproperty.py
def typedproperty(name, expected_type):
private_name = '_' + name
@property
def prop(self):
return getattr(self, private_name)
@prop.setter
def prop(self, value):
if not isinstance(value, expected_type):
raise TypeError(f'Expected {expected_type}')
setattr(self, private_name, value)
return prop
```
Now, try it out by defining a class like this:
```python
from typedproperty import typedproperty
class Stock(object):
name = typedproperty('name', str)
shares = typedproperty('shares', int)
price = typedproperty('price', float)
def __init__(self, name, shares, price):
self.name = name
self.shares = shares
self.price = price
```
Try creating an instance and verifying that type-checking works.
```python
>>> s = Stock('IBM', 50, 91.1)
>>> s.name
'IBM'
>>> s.shares = '100'
... should get a TypeError ...
>>>
```
### (b) Simplifying Function Calls
In the above example, users might find calls such as
`typedproperty('shares', int)` a bit verbose to type--especially if
they're repeated a lot. Add the following definitions to the
`typedproperty.py` file:
```python
String = lambda name: typedproperty(name, str)
Integer = lambda name: typedproperty(name, int)
Float = lambda name: typedproperty(name, float)
```
Now, rewrite the `Stock` class to use these functions instead:
```python
class Stock(object):
name = String('name')
shares = Integer('shares')
price = Float('price')
def __init__(self, name, shares, price):
self.name = name
self.shares = shares
self.price = price
```
Ah, that's a bit better. The main takeaway here is that closures and `lambda`
can often be used to simplify code and eliminate annoying repetition. This
is often good.
### (c) Putting it into practice
Rewrite the `Stock` class in the file `stock.py` so that it uses typed properties
as shown.
[Next](04_Function_decorators)

152
Notes/07_Advanced_Topics/04_Function_decorators.md Normal file
View File

@@ -0,0 +1,152 @@
# 7.4 Function Decorators
This section introduces the concept of a decorator. This is an advanced
topic for which we only scratch the surface.
### Logging Example
Consider a function.
```python
def add(x, y):
return x + y
```
Now, consider the function with some logging.
```python
def add(x, y):
print('Calling add')
return x + y
```
Now a second function also with some logging.
```python
def sub(x, y):
print('Calling sub')
return x - y
```
### Observation
*Observation: It's kind of repetitive.*
Writing programs where there is a lot of code replication is often really annoying.
They are tedious to write and hard to maintain.
Especially if you decide that you want to change how it works (i.e., a different kind of logging perhaps).
### Example continuation
Perhaps you can make *logging wrappers*.
```python
def logged(func):
def wrapper(*args, **kwargs):
print('Calling', func.__name__)
return func(*args, **kwargs)
return wrapper
```
Now use it.
```python
def add(x, y):
return x + y
logged_add = logged(add)
```
What happens when you call the function returned by `logged`?
```python
logged_add(3, 4) # You see the logging message appear
```
This example illustrates the process of creating a so-called *wrapper function*.
**A wrapper is a function that wraps another function with some extra bits of processing.**
```python
>>> logged_add(3, 4)
Calling add # Extra output. Added by the wrapper
7
>>>
```
*Note: The `logged()` function creates the wrapper and returns it as a result.*
## Decorators
Putting wrappers around functions is extremely common in Python.
So common, there is a special syntax for it.
```python
def add(x, y):
return x + y
add = logged(add)
# Special syntax
@logged
def add(x, y):
return x + y
```
The special syntax performs the same exact steps as shown above. A decorator is just new syntax.
It is said to *decorate* the function.
### Commentary
There are many more subtle details to decorators than what has been presented here.
For example, using them in classes. Or using multiple decorators with a function.
However, the previous example is a good illustration of how their use tends to arise.
## Exercises
### (a) A decorator for timing
If you define a function, its name and module are stored in the
`__name__` and `__module__` attributes. For example:
```python
>>> def add(x,y):
return x+y
>>> add.__name__
'add'
>>> add.__module__
'__main__'
>>>
```
In a file `timethis.py`, write a decorator function `timethis(func)`
that wraps a function with an extra layer of logic that prints out how
long it takes for a function to execute. To do this, you'll surround
the function with timing calls like this:
```python
start = time.time()
r = func(*args,**kwargs)
end = time.time()
print('%s.%s: %f' % (func.__module__, func.__name__, end-start))
```
Here is an example of how your decorator should work:
```python
>>> from timethis import timethis
>>> @timethis
def countdown(n):
while n > 0:
n -= 1
>>> countdown(10000000)
__main__.countdown : 0.076562
>>>
```
Discussion: This `@timethis` decorator can be placed in front of any
function definition. Thus, you might use it as a diagnostic tool for
performance tuning.
[Next](05_Decorated_methods)

260
Notes/07_Advanced_Topics/05_Decorated_methods.md Normal file
View File

@@ -0,0 +1,260 @@
# 7.5 Decorated Methods
This section discusses a few common decorators that are used in
combination with method definitions.
### Predefined Decorators
There are predefined decorators used to specify special kinds of methods in class definitions.
```python
class Foo(object):
def bar(self,a):
...
@staticmethod
def spam(a):
...
@classmethod
def grok(cls,a):
...
@property
def name(self):
...
```
Let's go one by one.
### Static Methods
`@staticmethod` is used to define a so-called *static* class methods (from C++/Java).
A static method is a function that is part of the class, but which does *not* operate on instances.
```python
class Foo(object):
@staticmethod
def bar(x):
print('x =', x)
>>> Foo.bar(2) x=2
>>>
```
Static methods are sometimes used to implement internal supporting code for a class.
For example, code to help manage created instances (memory management, system resources, persistence, locking, etc).
They're also used by certain design patterns (not discussed here).
### Class Methods
`@classmethod` is used to define class methods.
A class method is a method that receives the *class* object as the first parameter instead of the instance.
```python
class Foo(object):
def bar(self):
print(self)
@classmethod
def spam(cls):
print(cls)
>>> f = Foo()
>>> f.bar()
<__main__.Foo object at 0x971690> # The instance `f`
>>> Foo.spam()
<class '__main__.Foo'> # The class `Foo`
>>>
```
Class methods are most often used as a tool for defining alternate constructors.
```python
class Date(object):
def __init__(self,year,month,day):
self.year = year
self.month = month
self.day = day
@classmethod
def today(cls):
# Notice how the class is passed as an argument
tm = time.localtime()
# And used to create a new instance
return cls(tm.tm_year, tm.tm_mon, tm.tm_mday)
d = Date.today()
```
Class methods solve some tricky problems with features like inheritance.
```python
class Date(object):
...
@classmethod
def today(cls):
# Gets the correct class (e.g. `NewDate`)
tm = time.localtime()
return cls(tm.tm_year, tm.tm_mon, tm.tm_mday)
class NewDate(Date):
...
d = NewDate.today()
```
## Exercises
Start this exercise by defining a `Date` class. For example:
```
>>> class Date(object):
def __init__(self,year,month,day):
self.year = year
self.month = month
self.day = day
>>> d = Date(2010, 4, 13)
>>> d.year, d.month, d.day
(2010, 4, 13)
>>>
```
### (a) Class Methods
A common use of class methods is to provide alternate constructors
(epecially since Python doesn't support overloaded methods). Modify
the `Date` class to have a class method `today()` that creates a date
from today's date.
```python
>>> import time
>>> class Date(object):
def __init__(self,year,month,day):
self.year = year
self.month = month
self.day = day
@classmethod
def today(cls):
t = time.localtime()
return cls(t.tm_year, t.tm_mon, t.tm_mday)
>>> d = Date.today()
>>> d.year, d.month, d.day
... output varies. Should be today ...
>>>
```
One reason you should use class methods for this is that they work
with inheritance. For example, try this:
```python
>>> class CustomDate(Date):
def yow(self):
print('Yow!')
>>> d = CustomDate.today()
<__main__.CustomDate object at 0x10923d400>
>>> d.yow()
Yow!
>>>
```
### (b) Class Methods in Practice
In your `report.py` and `portfolio.py` files, the creation of a `Portfolio`
object is a bit muddled. For example, the `report.py` program has code like this:
```python
def read_portfolio(filename, **opts):
'''
Read a stock portfolio file into a list of dictionaries with keys
name, shares, and price.
'''
with open(filename) as lines:
portdicts = fileparse.parse_csv(lines,
select=['name','shares','price'],
types=[str,int,float],
**opts)
portfolio = [ Stock(**d) for d in portdicts ]
return Portfolio(portfolio)
```
and the `portfolio.py` file defines `Portfolio()` with an odd initializer
like this:
```python
class Portfolio(object):
def __init__(self, holdings):
self.holdings = holdings
...
```
Frankly, the chain of responsibility is all a bit confusing because the
code is scattered. If a `Portfolio` class is supposed to contain
a list of `Stock` instances, maybe you should change the class to be a bit more clear.
Like this:
```python
# portfolio.py
import stock
class Portfolio(object):
def __init__(self):
self.holdings = []
def append(self, holding):
if not isinstance(holding, stock.Stock):
raise TypeError('Expected a Stock instance')
self.holdings.append(holding)
...
```
If you want to read a portfolio from a CSV file, maybe you should make a
class method for it:
```python
# portfolio.py
import fileparse
import stock
class Portfolio(object):
def __init__(self):
self.holdings = []
def append(self, holding):
if not isinstance(holding, stock.Stock):
raise TypeError('Expected a Stock instance')
self.holdings.append(holding)
@classmethod
def from_csv(cls, lines, **opts):
self = cls()
portdicts = fileparse.parse_csv(lines,
select=['name','shares','price'],
types=[str,int,float],
**opts)
for d in portdicts:
self.append(stock.Stock(**d))
return self
```
To use this new Portfolio class, you can now write code like this:
```
>>> from portfolio import Portfolio
>>> with open('Data/portfolio.csv') as lines:
... port = Portfolio.from_csv(lines)
...
>>>
```
Make these changes to the `Portfolio` class and modify the `report.py`
code to use the class method.