zev/practical-python
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Editing

Browse Source
This commit is contained in:
David Beazley
2020-05-28 10:22:32 -05:00
parent a5cae9cdc2
commit 7a1cccb847
6 changed files with 268 additions and 170 deletions
Download Patch File Download Diff File Expand all files Collapse all files

77
Notes/03_Program_organization/01_Script.md
View File

@@ -1,3 +1,5 @@
[Contents](../Contents) \| [Previous (2.7 Object Model)](../02_Working_with_data/07_Objects) \| [Next (3.2 More on Functions)](02_More_functions)
# 3.1 Scripting # 3.1 Scripting
In this part we look more closely at the practice of writing Python In this part we look more closely at the practice of writing Python
@@ -16,7 +18,7 @@ statement3
... ...
``` ```
We have been writing scripts to this point. We have mostly been writing scripts to this point.
### A Problem ### A Problem
@@ -28,7 +30,7 @@ organized.
### Defining Things ### Defining Things
You must always define things before they get used later on in a program. Names must always be defined before they get used later.
```python ```python
def square(x): def square(x):
@@ -41,11 +43,12 @@ z = square(b) # Requires `square` and `b` to be defined
``` ```
**The order is important.** **The order is important.**
You almost always put the definitions of variables an functions near the beginning. You almost always put the definitions of variables and functions near the top.
### Defining Functions ### Defining Functions
It is a good idea to put all of the code related to a single *task* all in one place. It is a good idea to put all of the code related to a single *task* all in one place.
Use a function.
```python ```python
def read_prices(filename): def read_prices(filename):
@@ -85,7 +88,7 @@ def foo():
help(math) help(math)
``` ```
There are no *special* statements in Python. There are no *special* statements in Python (which makes it easy to remember).
### Function Definition ### Function Definition
@@ -106,13 +109,14 @@ def foo(x):
bar(x) bar(x)
``` ```
Functions must only be defined before they are actually *used* (or called) during program execution. Functions must only be defined prior to actually being *used* (or called) during program execution.
```python ```python
foo(3) # foo must be defined already foo(3) # foo must be defined already
``` ```
Stylistically, it is probably more common to see functions defined in a *bottom-up* fashion. Stylistically, it is probably more common to see functions defined in
a *bottom-up* fashion.
### Bottom-up Style ### Bottom-up Style
@@ -137,24 +141,26 @@ def spam(x):
spam(42) # Code that uses the functions appears at the end spam(42) # Code that uses the functions appears at the end
``` ```
Later functions build upon earlier functions. Later functions build upon earlier functions. Again, this is only
a point of style. The only thing that matters in the above program
is that the call to `spam(42)` go last.
### Function Design ### Function Design
Ideally, functions should be a *black box*. Ideally, functions should be a *black box*.
They should only operate on passed inputs and avoid global variables They should only operate on passed inputs and avoid global variables
and mysterious side-effects. Main goals: *Modularity* and *Predictability*. and mysterious side-effects. Your main goals: *Modularity* and *Predictability*.
### Doc Strings ### Doc Strings
A good practice is to include documentations in the form of It's good practice to include documentation in the form of a
doc-strings. Doc-strings are strings written immediately after the doc-string. Doc-strings are strings written immediately after the
name of the function. They feed `help()`, IDEs and other tools. name of the function. They feed `help()`, IDEs and other tools.
```python ```python
def read_prices(filename): def read_prices(filename):
''' '''
Read prices from a CSV file of name,price Read prices from a CSV file of name,price data
''' '''
prices = {} prices = {}
with open(filename) as f: with open(filename) as f:
@@ -164,14 +170,19 @@ def read_prices(filename):
return prices return prices
``` ```
A good practice for doc strings is to write a short one sentence
summary of what the function does. If more information is needed,
include a short example of usage along with a more detailed
description of the arguments.
### Type Annotations ### Type Annotations
You can also add some optional type annotations to your function definitions. You can also add optional type hints to function definitions.
```python ```python
def read_prices(filename: str) -> dict: def read_prices(filename: str) -> dict:
''' '''
Read prices from a CSV file of name,price Read prices from a CSV file of name,price data
''' '''
prices = {} prices = {}
with open(filename) as f: with open(filename) as f:
@@ -181,13 +192,15 @@ def read_prices(filename: str) -> dict:
return prices return prices
``` ```
These do nothing. It is purely informational. The hints do nothing operationally. They are purely informational.
They may be used by IDEs, code checkers, etc. However, they may be used by IDEs, code checkers, and other tools
to do more.
## Exercises ## Exercises
In section 2, you wrote a program called `report.py` that printed out a report showing the performance of a stock portfolio. In section 2, you wrote a program called `report.py` that printed out
This program consisted of some functions. For example: a report showing the performance of a stock portfolio. This program
consisted of some functions. For example:
```python ```python
# report.py # report.py
@@ -215,8 +228,9 @@ def read_portfolio(filename):
... ...
``` ```
However, there were also portions of the program that just performed a series of scripted calculations. However, there were also portions of the program that just performed a
This code appeared near the end of the program. For example: series of scripted calculations. This code appeared near the end of
the program. For example:
```python ```python
... ...
@@ -231,7 +245,8 @@ for row in report:
... ...
``` ```
In this exercise, we’re going take this program and organize it a little more strongly around the use of functions. In this exercise, we’re going take this program and organize it a
little more strongly around the use of functions.
### Exercise 3.1: Structuring a program as a collection of functions ### Exercise 3.1: Structuring a program as a collection of functions
@@ -242,10 +257,12 @@ functions. Specifically:
* Create a function `print_report(report)` that prints out the report. * Create a function `print_report(report)` that prints out the report.
* Change the last part of the program so that it is nothing more than a series of function calls and no other computation. * Change the last part of the program so that it is nothing more than a series of function calls and no other computation.
### Exercise 3.2: Creating a function for program execution ### Exercise 3.2: Creating a top-level function for program execution
Take the last part of your program and package it into a single function `portfolio_report(portfolio_filename, prices_filename)`. Take the last part of your program and package it into a single
Have the function work so that the following function call creates the report as before: function `portfolio_report(portfolio_filename, prices_filename)`.
Have the function work so that the following function call creates the
report as before:
```python ```python
portfolio_report('Data/portfolio.csv', 'Data/prices.csv') portfolio_report('Data/portfolio.csv', 'Data/prices.csv')
@@ -256,8 +273,9 @@ of function definitions followed by a single function call to
`portfolio_report()` at the very end (which executes all of the steps `portfolio_report()` at the very end (which executes all of the steps
involved in the program). involved in the program).
By turning your program into a single function, it becomes easy to run it on different inputs. By turning your program into a single function, it becomes easy to run
For example, try these statements interactively after running your program: it on different inputs. For example, try these statements
interactively after running your program:
```python ```python
>>> portfolio_report('Data/portfolio2.csv', 'Data/prices.csv') >>> portfolio_report('Data/portfolio2.csv', 'Data/prices.csv')
@@ -272,4 +290,13 @@ For example, try these statements interactively after running your program:
>>> >>>
``` ```
### Commentary
Python makes it very easy to write relatively unstructured scripting code
where you just have a file with a sequence of statements in it. In the
big picture, it's almost always better to utilize functions whenever
you can. At some point, that script is going to grow and you'll wish
you had a bit more organization. Also, a little known fact is that Python
runs a bit faster if you use functions.
[Contents](../Contents) \| [Previous (2.7 Object Model)](../02_Working_with_data/07_Objects) \| [Next (3.2 More on Functions)](02_More_functions) [Contents](../Contents) \| [Previous (2.7 Object Model)](../02_Working_with_data/07_Objects) \| [Next (3.2 More on Functions)](02_More_functions)

135
Notes/03_Program_organization/02_More_functions.md
View File

@@ -1,6 +1,10 @@
[Contents](../Contents) \| [Previous (3.1 Scripting)](01_Script) \| [Next (3.3 Error Checking)](03_Error_checking)
# 3.2 More on Functions # 3.2 More on Functions
This section fills in a few more details about how functions work and are defined. Although functions were introduced earlier, very few details were provided on how
they actually work at a deeper level. This section aims to fill in some gaps
and discuss matters such as calling conventions, scoping rules, and more.
### Calling a Function ### Calling a Function
@@ -25,7 +29,8 @@ prices = read_prices(filename='prices.csv', debug=True)
### Default Arguments ### Default Arguments
Sometimes you want an optional argument. Sometimes you want an argument to be optional. If so, assign a default value
in the function definition.
```python ```python
def read_prices(filename, debug=False): def read_prices(filename, debug=False):
@@ -53,7 +58,8 @@ parse_data(data, debug=True)
parse_data(data, debug=True, ignore_errors=True) parse_data(data, debug=True, ignore_errors=True)
``` ```
Keyword arguments improve code clarity. In most cases, keyword arguments improve code clarity--especially for arguments that
serve as flags or which are related to optional features.
### Design Best Practices ### Design Best Practices
@@ -67,7 +73,7 @@ d = read_prices('prices.csv', debug=True)
Python development tools will show the names in help features and documentation. Python development tools will show the names in help features and documentation.
### Return Values ### Returning Values
The `return` statement returns a value The `return` statement returns a value
@@ -76,7 +82,7 @@ def square(x):
return x * x return x * x
``` ```
If no return value or `return` not specified, `None` is returned. If no return value is given or `return` is missing, `None` is returned.
```python ```python
def bar(x): def bar(x):
@@ -94,8 +100,8 @@ b = foo(4) # b = None
### Multiple Return Values ### Multiple Return Values
Functions can only return one value. Functions can only return one value. However, a function may return
However, a function may return multiple values by returning a tuple. multiple values by returning them in a tuple.
```python ```python
def divide(a,b): def divide(a,b):
@@ -124,18 +130,19 @@ def foo():
``` ```
Variables assignments occur outside and inside function definitions. Variables assignments occur outside and inside function definitions.
Variables defined outside are "global". Variables inside a function are "local". Variables defined outside are "global". Variables inside a function
are "local".
### Local Variables ### Local Variables
Variables inside functions are private. Variables assigned inside functions are private.
```python ```python
def read_portfolio(filename): def read_portfolio(filename):
portfolio = [] portfolio = []
for line in open(filename): for line in open(filename):
fields = line.split() fields = line.split(',')
s = (fields[0],int(fields[1]),float(fields[2])) s = (fields[0], int(fields[1]), float(fields[2]))
portfolio.append(s) portfolio.append(s)
return portfolio return portfolio
``` ```
@@ -143,8 +150,8 @@ def read_portfolio(filename):
In this example, `filename`, `portfolio`, `line`, `fields` and `s` are local variables. In this example, `filename`, `portfolio`, `line`, `fields` and `s` are local variables.
Those variables are not retained or accessible after the function call. Those variables are not retained or accessible after the function call.
```pycon ```python
>>> stocks = read_portfolio('stocks.dat') >>> stocks = read_portfolio('portfolio.csv')
>>> fields >>> fields
Traceback (most recent call last): Traceback (most recent call last):
File "<stdin>", line 1, in ? File "<stdin>", line 1, in ?
@@ -152,11 +159,12 @@ NameError: name 'fields' is not defined
>>> >>>
``` ```
They also can't conflict with variables found elsewhere. Locals also can't conflict with variables found elsewhere.
### Global Variables ### Global Variables
Functions can freely access the values of globals. Functions can freely access the values of globals defined in the same
file.
```python ```python
name = 'Dave' name = 'Dave'
@@ -185,20 +193,24 @@ If you must modify a global variable you must declare it as such.
```python ```python
name = 'Dave' name = 'Dave'
def spam(): def spam():
global name global name
name = 'Guido' # Changes the global name above name = 'Guido' # Changes the global name above
``` ```
The global declaration must appear before its use. Having seen this, The global declaration must appear before its use and the corresponding
know that it is considered poor form. In fact, try to avoid entirely variable must exist in the same file as the function. Having seen this,
know that it is considered poor form. In fact, try to avoid `global` entirely
if you can. If you need a function to modify some kind of state outside if you can. If you need a function to modify some kind of state outside
of the function, it's better to use a class instead (more on this later). of the function, it's better to use a class instead (more on this later).
### Argument Passing ### Argument Passing
When you call a function, the argument variables are names for passed values. When you call a function, the argument variables are names that refer
If mutable data types are passed (e.g. lists, dicts), they can be modified *in-place*. to the passed values. These values are NOT copies (see [section
2.7](../02_Working_with_data/07_Objects)). If mutable data types are
passed (e.g. lists, dicts), they can be modified *in-place*.
```python ```python
def foo(items): def foo(items):
@@ -213,7 +225,8 @@ print(a) # [1, 2, 3, 42]
### Reassignment vs Modifying ### Reassignment vs Modifying
Make sure you understand the subtle difference between modifying a value and reassigning a variable name. Make sure you understand the subtle difference between modifying a
value and reassigning a variable name.
```python ```python
def foo(items): def foo(items):
@@ -225,19 +238,22 @@ print(a) # [1, 2, 3, 42]
# VS # VS
def bar(items): def bar(items):
items = [4,5,6] # Reassigns `items` variable items = [4,5,6] # Changes local `items` variable to point to a different object
b = [1, 2, 3] b = [1, 2, 3]
bar(b) bar(b)
print(b) # [1, 2, 3] print(b) # [1, 2, 3]
``` ```
*Reminder: Variable assignment never overwrites memory. The name is simply bound to a new value.* *Reminder: Variable assignment never overwrites memory. The name is merely bound to a new value.*
## Exercises ## Exercises
This exercise involves a lot of steps and putting concepts together from past exercises. This set of exercises have you implement what is, perhaps, the most
The final solution is only about 25 lines of code, but take your time and make sure you understand each part. powerful and difficult part of the course. There are a lot of steps
and many concepts from past exercises are put together all at once.
The final solution is only about 25 lines of code, but take your time
and make sure you understand each part.
A central part of your `report.py` program focuses on the reading of A central part of your `report.py` program focuses on the reading of
CSV files. For example, the function `read_portfolio()` reads a file CSV files. For example, the function `read_portfolio()` reads a file
@@ -251,12 +267,13 @@ If you were doing a lot of file parsing for real, you’d probably want
to clean some of this up and make it more general purpose. That's to clean some of this up and make it more general purpose. That's
our goal. our goal.
Start this exercise by creating a new file called `fileparse.py`. This is where we will be doing our work. Start this exercise by creating a new file called
`Work/fileparse.py`. This is where we will be doing our work.
### Exercise 3.3: Reading CSV Files ### Exercise 3.3: Reading CSV Files
To start, let’s just focus on the problem of reading a CSV file into a To start, let’s just focus on the problem of reading a CSV file into a
list of dictionaries. In the file `fileparse.py`, define a simple list of dictionaries. In the file `fileparse.py`, define a
function that looks like this: function that looks like this:
```python ```python
@@ -290,20 +307,23 @@ Try it out:
Hint: `python3 -i fileparse.py`. Hint: `python3 -i fileparse.py`.
```pycon ```python
>>> portfolio = parse_csv('Data/portfolio.csv') >>> portfolio = parse_csv('Data/portfolio.csv')
>>> portfolio >>> portfolio
[{'price': '32.20', 'name': 'AA', 'shares': '100'}, {'price': '91.10', 'name': 'IBM', 'shares': '50'}, {'price': '83.44', 'name': 'CAT', 'shares': '150'}, {'price': '51.23', 'name': 'MSFT', 'shares': '200'}, {'price': '40.37', 'name': 'GE', 'shares': '95'}, {'price': '65.10', 'name': 'MSFT', 'shares': '50'}, {'price': '70.44', 'name': 'IBM', 'shares': '100'}] [{'price': '32.20', 'name': 'AA', 'shares': '100'}, {'price': '91.10', 'name': 'IBM', 'shares': '50'}, {'price': '83.44', 'name': 'CAT', 'shares': '150'}, {'price': '51.23', 'name': 'MSFT', 'shares': '200'}, {'price': '40.37', 'name': 'GE', 'shares': '95'}, {'price': '65.10', 'name': 'MSFT', 'shares': '50'}, {'price': '70.44', 'name': 'IBM', 'shares': '100'}]
>>> >>>
``` ```
This is great except that you can’t do any kind of useful calculation with the data because everything is represented as a string. This is good except that you can’t do any kind of useful calculation
We’ll fix this shortly, but let’s keep building on it. with the data because everything is represented as a string. We’ll
fix this shortly, but let’s keep building on it.
### Exercise 3.4: Building a Column Selector ### Exercise 3.4: Building a Column Selector
In many cases, you’re only interested in selected columns from a CSV file, not all of the data. In many cases, you’re only interested in selected columns from a CSV
Modify the `parse_csv()` function so that it optionally allows user-specified columns to be picked out as follows: file, not all of the data. Modify the `parse_csv()` function so that
it optionally allows user-specified columns to be picked out as
follows:
```python ```python
>>> # Read all of the data >>> # Read all of the data
@@ -311,7 +331,7 @@ Modify the `parse_csv()` function so that it optionally allows user-specified co
>>> portfolio >>> portfolio
[{'price': '32.20', 'name': 'AA', 'shares': '100'}, {'price': '91.10', 'name': 'IBM', 'shares': '50'}, {'price': '83.44', 'name': 'CAT', 'shares': '150'}, {'price': '51.23', 'name': 'MSFT', 'shares': '200'}, {'price': '40.37', 'name': 'GE', 'shares': '95'}, {'price': '65.10', 'name': 'MSFT', 'shares': '50'}, {'price': '70.44', 'name': 'IBM', 'shares': '100'}] [{'price': '32.20', 'name': 'AA', 'shares': '100'}, {'price': '91.10', 'name': 'IBM', 'shares': '50'}, {'price': '83.44', 'name': 'CAT', 'shares': '150'}, {'price': '51.23', 'name': 'MSFT', 'shares': '200'}, {'price': '40.37', 'name': 'GE', 'shares': '95'}, {'price': '65.10', 'name': 'MSFT', 'shares': '50'}, {'price': '70.44', 'name': 'IBM', 'shares': '100'}]
>>> # Read some of the data >>> # Read only some of the data
>>> shares_held = parse_csv('portfolio.csv', select=['name','shares']) >>> shares_held = parse_csv('portfolio.csv', select=['name','shares'])
>>> shares_held >>> shares_held
[{'name': 'AA', 'shares': '100'}, {'name': 'IBM', 'shares': '50'}, {'name': 'CAT', 'shares': '150'}, {'name': 'MSFT', 'shares': '200'}, {'name': 'GE', 'shares': '95'}, {'name': 'MSFT', 'shares': '50'}, {'name': 'IBM', 'shares': '100'}] [{'name': 'AA', 'shares': '100'}, {'name': 'IBM', 'shares': '50'}, {'name': 'CAT', 'shares': '150'}, {'name': 'MSFT', 'shares': '200'}, {'name': 'GE', 'shares': '95'}, {'name': 'MSFT', 'shares': '50'}, {'name': 'IBM', 'shares': '100'}]
@@ -358,17 +378,18 @@ def parse_csv(filename, select=None):
return records return records
``` ```
There are a number of tricky bits to this part. Probably the most important one is the mapping of the column selections to row indices. There are a number of tricky bits to this part. Probably the most
important one is the mapping of the column selections to row indices.
For example, suppose the input file had the following headers: For example, suppose the input file had the following headers:
```pycon ```python
>>> headers = ['name', 'date', 'time', 'shares', 'price'] >>> headers = ['name', 'date', 'time', 'shares', 'price']
>>> >>>
``` ```
Now, suppose the selected columns were as follows: Now, suppose the selected columns were as follows:
```pycon ```python
>>> select = ['name', 'shares'] >>> select = ['name', 'shares']
>>> >>>
``` ```
@@ -376,7 +397,7 @@ Now, suppose the selected columns were as follows:
To perform the proper selection, you have to map the selected column names to column indices in the file. To perform the proper selection, you have to map the selected column names to column indices in the file.
That’s what this step is doing: That’s what this step is doing:
```pycon ```python
>>> indices = [headers.index(colname) for colname in select ] >>> indices = [headers.index(colname) for colname in select ]
>>> indices >>> indices
[0, 3] [0, 3]
@@ -386,7 +407,7 @@ That’s what this step is doing:
In other words, "name" is column 0 and "shares" is column 3. In other words, "name" is column 0 and "shares" is column 3.
When you read a row of data from the file, the indices are used to filter it: When you read a row of data from the file, the indices are used to filter it:
```pycon ```python
>>> row = ['AA', '6/11/2007', '9:50am', '100', '32.20' ] >>> row = ['AA', '6/11/2007', '9:50am', '100', '32.20' ]
>>> row = [ row[index] for index in indices ] >>> row = [ row[index] for index in indices ]
>>> row >>> row
@@ -396,10 +417,10 @@ When you read a row of data from the file, the indices are used to filter it:
### Exercise 3.5: Performing Type Conversion ### Exercise 3.5: Performing Type Conversion
Modify the `parse_csv()` function so that it optionally allows type-conversions to be applied to the returned data. Modify the `parse_csv()` function so that it optionally allows
For example: type-conversions to be applied to the returned data. For example:
```pycon ```python
>>> portfolio = parse_csv('Data/portfolio.csv', types=[str, int, float]) >>> portfolio = parse_csv('Data/portfolio.csv', types=[str, int, float])
>>> portfolio >>> portfolio
[{'price': 32.2, 'name': 'AA', 'shares': 100}, {'price': 91.1, 'name': 'IBM', 'shares': 50}, {'price': 83.44, 'name': 'CAT', 'shares': 150}, {'price': 51.23, 'name': 'MSFT', 'shares': 200}, {'price': 40.37, 'name': 'GE', 'shares': 95}, {'price': 65.1, 'name': 'MSFT', 'shares': 50}, {'price': 70.44, 'name': 'IBM', 'shares': 100}] [{'price': 32.2, 'name': 'AA', 'shares': 100}, {'price': 91.1, 'name': 'IBM', 'shares': 50}, {'price': 83.44, 'name': 'CAT', 'shares': 150}, {'price': 51.23, 'name': 'MSFT', 'shares': 200}, {'price': 40.37, 'name': 'GE', 'shares': 95}, {'price': 65.1, 'name': 'MSFT', 'shares': 50}, {'price': 70.44, 'name': 'IBM', 'shares': 100}]
@@ -410,8 +431,8 @@ For example:
>>> >>>
``` ```
You already explored this in [Exercise 2.24](../02_Working_with_data/07_Objects). You'll need to insert the You already explored this in [Exercise 2.24](../02_Working_with_data/07_Objects).
following fragment of code into your solution: You'll need to insert the following fragment of code into your solution:
```python ```python
... ...
@@ -420,7 +441,7 @@ if types:
... ...
``` ```
### Exercise 3.6: Working with Headers ### Exercise 3.6: Working without Headers
Some CSV files don’t include any header information. Some CSV files don’t include any header information.
For example, the file `prices.csv` looks like this: For example, the file `prices.csv` looks like this:
@@ -433,8 +454,8 @@ For example, the file `prices.csv` looks like this:
... ...
``` ```
Modify the `parse_csv()` function so that it can work with such files by creating a list of tuples instead. Modify the `parse_csv()` function so that it can work with such files
For example: by creating a list of tuples instead. For example:
```python ```python
>>> prices = parse_csv('Data/prices.csv', types=[str,float], has_headers=False) >>> prices = parse_csv('Data/prices.csv', types=[str,float], has_headers=False)
@@ -450,8 +471,10 @@ column names to use for keys.
### Exercise 3.7: Picking a different column delimitier ### Exercise 3.7: Picking a different column delimitier
Although CSV files are pretty common, it’s also possible that you could encounter a file that uses a different column separator such as a tab or space. Although CSV files are pretty common, it’s also possible that you
For example, the file `Data/portfolio.dat` looks like this: could encounter a file that uses a different column separator such as
a tab or space. For example, the file `Data/portfolio.dat` looks like
this:
```csv ```csv
name shares price name shares price
@@ -464,26 +487,30 @@ name shares price
"IBM" 100 70.44 "IBM" 100 70.44
``` ```
The `csv.reader()` function allows a different delimiter to be given as follows: The `csv.reader()` function allows a different column delimiter to be given as follows:
```python ```python
rows = csv.reader(f, delimiter=' ') rows = csv.reader(f, delimiter=' ')
``` ```
Modify your `parse_csv()` function so that it also allows the delimiter to be changed. Modify your `parse_csv()` function so that it also allows the
delimiter to be changed.
For example: For example:
```pycon ```python
>>> portfolio = parse_csv('Data/portfolio.dat', types=[str, int, float], delimiter=' ') >>> portfolio = parse_csv('Data/portfolio.dat', types=[str, int, float], delimiter=' ')
>>> portfolio >>> portfolio
[{'price': '32.20', 'name': 'AA', 'shares': '100'}, {'price': '91.10', 'name': 'IBM', 'shares': '50'}, {'price': '83.44', 'name': 'CAT', 'shares': '150'}, {'price': '51.23', 'name': 'MSFT', 'shares': '200'}, {'price': '40.37', 'name': 'GE', 'shares': '95'}, {'price': '65.10', 'name': 'MSFT', 'shares': '50'}, {'price': '70.44', 'name': 'IBM', 'shares': '100'}] [{'price': '32.20', 'name': 'AA', 'shares': '100'}, {'price': '91.10', 'name': 'IBM', 'shares': '50'}, {'price': '83.44', 'name': 'CAT', 'shares': '150'}, {'price': '51.23', 'name': 'MSFT', 'shares': '200'}, {'price': '40.37', 'name': 'GE', 'shares': '95'}, {'price': '65.10', 'name': 'MSFT', 'shares': '50'}, {'price': '70.44', 'name': 'IBM', 'shares': '100'}]
>>> >>>
``` ```
If you’ve made it this far, you’ve created a nice library function that’s genuinely useful. ### Commentary
You can use it to parse arbitrary CSV files, select out columns of
interest, perform type conversions, without having to worry too much If you’ve made it this far, you’ve created a nice library function
about the inner workings of files or the `csv` module. that’s genuinely useful. You can use it to parse arbitrary CSV files,
select out columns of interest, perform type conversions, without
having to worry too much about the inner workings of files or the
`csv` module.
[Contents](../Contents) \| [Previous (3.1 Scripting)](01_Script) \| [Next (3.3 Error Checking)](03_Error_checking) [Contents](../Contents) \| [Previous (3.1 Scripting)](01_Script) \| [Next (3.3 Error Checking)](03_Error_checking)

86
Notes/03_Program_organization/03_Error_checking.md
View File

@@ -1,11 +1,15 @@
[Contents](../Contents) \| [Previous (3.2 More on Functions)](02_More_functions) \| [Next (3.4 Modules)](04_Modules)
# 3.3 Error Checking # 3.3 Error Checking
This section discusses some aspects of error checking and exception handling. Although exceptions were introduced earlier, this section fills in some additional
details about error checking and exception handling.
### How programs fail ### How programs fail
Python performs no checking or validation of function argument types or values. Python performs no checking or validation of function argument types
A function will work on any data that is compatible with the statements in the function. or values. A function will work on any data that is compatible with
the statements in the function.
```python ```python
def add(x, y): def add(x, y):
@@ -16,7 +20,7 @@ add('Hello', 'World') # 'HelloWorld'
add('3', '4') # '34' add('3', '4') # '34'
``` ```
If there are errors in a function, they will show up at run time (as an exception). If there are errors in a function, they appear at run time (as an exception).
```python ```python
def add(x, y): def add(x, y):
@@ -38,8 +42,8 @@ Exceptions are used to signal errors.
To raise an exception yourself, use `raise` statement. To raise an exception yourself, use `raise` statement.
```python ```python
if name not in names: if name not in authorized:
raise RuntimeError('Name not found') raise RuntimeError(f'{name} not authorized')
``` ```
To catch an exception use `try-except`. To catch an exception use `try-except`.
@@ -78,7 +82,8 @@ def foo():
foo() foo()
``` ```
To handle the exception, use the `except` block. You can add any statements you want to handle the error. To handle the exception, put statements in the `except` block. You can add any
statements you want to handle the error.
```python ```python
def grok(): ... def grok(): ...
@@ -95,7 +100,8 @@ def bar():
bar() bar()
``` ```
After handling, execution resumes with the first statement after the `try-except`. After handling, execution resumes with the first statement after the
`try-except`.
```python ```python
def grok(): ... def grok(): ...
@@ -117,8 +123,10 @@ bar()
### Built-in Exceptions ### Built-in Exceptions
There are about two-dozen built-in exceptions. There are about two-dozen built-in exceptions. Usually the name of
This is not an exhaustive list. Check the documentation for more. the exception is indicative of what's wrong (e.g., a `ValueError` is
raised because you supplied a bad value). This is not an
exhaustive list. Check the documentation for more.
```python ```python
ArithmeticError ArithmeticError
@@ -141,22 +149,23 @@ ValueError
### Exception Values ### Exception Values
Most exceptions have an associated value. It contains more information about what's wrong. Exceptions have an associated value. It contains more specific
information about what's wrong.
```python ```python
raise RuntimeError('Invalid user name') raise RuntimeError('Invalid user name')
``` ```
This value is passed to the variable supplied in `except`. This value is part of the exception instance that's placed in the variable supplied to `except`.
```python ```python
try: try:
... ...
except RuntimeError as e: # `e` holds the value raised except RuntimeError as e: # `e` holds the exception raised
... ...
``` ```
The value is an instance of the exception type. However, it often looks like a string when `e` is an instance of the exception type. However, it often looks like a string when
printed. printed.
```python ```python
@@ -166,7 +175,7 @@ except RuntimeError as e:
### Catching Multiple Errors ### Catching Multiple Errors
You can catch different kinds of exceptions with multiple `except` blocks. You can catch different kinds of exceptions using multiple `except` blocks.
```python ```python
try: try:
@@ -181,7 +190,7 @@ except KeyboardInterrupt as e:
... ...
``` ```
Alternatively, if the block to handle them is the same, you can group them: Alternatively, if the statements to handle them is the same, you can group them:
```python ```python
try: try:
@@ -197,12 +206,12 @@ To catch any exception, use `Exception` like this:
```python ```python
try: try:
... ...
except Exception: except Exception: # DANGER. See below
print('An error occurred') print('An error occurred')
``` ```
In general, writing code like that is a bad idea because you'll have no idea In general, writing code like that is a bad idea because you'll have
why it failed. no idea why it failed.
### Wrong Way to Catch Errors ### Wrong Way to Catch Errors
@@ -215,13 +224,13 @@ except Exception:
print('Computer says no') print('Computer says no')
``` ```
This swallows all possible errors. It may make it impossible to debug This catches all possible errors and it may make it impossible to debug
when the code is failing for some reason you didn't expect at all when the code is failing for some reason you didn't expect at all
(e.g. uninstalled Python module, etc.). (e.g. uninstalled Python module, etc.).
### Somewhat Better Approach ### Somewhat Better Approach
This is a more sane approach. If you're going to catch all errors, this is a more sane approach.
```python ```python
try: try:
@@ -234,9 +243,9 @@ It reports a specific reason for failure. It is almost always a good
idea to have some mechanism for viewing/reporting errors when you idea to have some mechanism for viewing/reporting errors when you
write code that catches all possible exceptions. write code that catches all possible exceptions.
In general though, it's better to catch the error more narrowly. Only In general though, it's better to catch the error as narrowly as is
catch the errors you can actually deal with. Let other errors pass to reasonable. Only catch the errors you can actually handle. Let
other code. other errors pass by--maybe some other code can handle them.
### Reraising an Exception ### Reraising an Exception
@@ -250,7 +259,8 @@ except Exception as e:
raise raise
``` ```
It allows you to take action (e.g. logging) and pass the error on to the caller. This allows you to take action (e.g. logging) and pass the error on to
the caller.
### Exception Best Practices ### Exception Best Practices
@@ -261,7 +271,8 @@ and sanely keep going.
### `finally` statement ### `finally` statement
It specifies code that must fun regardless of whether or not an exception occurs. It specifies code that must run regardless of whether or not an
exception occurs.
```python ```python
lock = Lock() lock = Lock()
@@ -273,11 +284,11 @@ finally:
lock.release() # this will ALWAYS be executed. With and without exception. lock.release() # this will ALWAYS be executed. With and without exception.
``` ```
Comonly used to properly manage resources (especially locks, files, etc.). Commonly used to safely manage resources (especially locks, files, etc.).
### `with` statement ### `with` statement
In modern code, `try-finally` often replaced with the `with` statement. In modern code, `try-finally` is often replaced with the `with` statement.
```python ```python
lock = Lock() lock = Lock()
@@ -296,8 +307,9 @@ with open(filename) as f:
# File closed # File closed
``` ```
It defines a usage *context* for a resource. When execution leaves that context, `with` defines a usage *context* for a resource. When execution
resources are released. `with` only works with certain objects. leaves that context, resources are released. `with` only works with
certain objects that have been specifically programmed to support it.
## Exercises ## Exercises
@@ -308,8 +320,7 @@ user-specified columns to be selected, but that only works if the
input data file has column headers. input data file has column headers.
Modify the code so that an exception gets raised if both the `select` Modify the code so that an exception gets raised if both the `select`
and `has_headers=False` arguments are passed. and `has_headers=False` arguments are passed. For example:
For example:
```python ```python
>>> parse_csv('Data/prices.csv', select=['name','price'], has_headers=False) >>> parse_csv('Data/prices.csv', select=['name','price'], has_headers=False)
@@ -335,6 +346,7 @@ in a non-sensical mode (e.g., using a feature that requires column
headers, but simultaneously specifying that there are no headers). headers, but simultaneously specifying that there are no headers).
This indicates a programming error on the part of the calling code. This indicates a programming error on the part of the calling code.
Checking for cases that "aren't supposed to happen" is often a good idea.
### Exercise 3.9: Catching exceptions ### Exercise 3.9: Catching exceptions
@@ -357,9 +369,9 @@ Modify the `parse_csv()` function to catch all `ValueError` exceptions
generated during record creation and print a warning message for rows generated during record creation and print a warning message for rows
that can’t be converted. that can’t be converted.
The message should include the row number and information about the reason why it failed. The message should include the row number and information about the
To test your function, try reading the file `Data/missing.csv` above. reason why it failed. To test your function, try reading the file
For example: `Data/missing.csv` above. For example:
```python ```python
>>> portfolio = parse_csv('Data/missing.csv', types=[str, int, float]) >>> portfolio = parse_csv('Data/missing.csv', types=[str, int, float])
@@ -375,8 +387,8 @@ Row 7: Reason invalid literal for int() with base 10: ''
### Exercise 3.10: Silencing Errors ### Exercise 3.10: Silencing Errors
Modify the `parse_csv()` function so that parsing error messages can be silenced if explicitly desired by the user. Modify the `parse_csv()` function so that parsing error messages can
For example: be silenced if explicitly desired by the user. For example:
```python ```python
>>> portfolio = parse_csv('Data/missing.csv', types=[str,int,float], silence_errors=True) >>> portfolio = parse_csv('Data/missing.csv', types=[str,int,float], silence_errors=True)

66
Notes/03_Program_organization/04_Modules.md
View File

@@ -1,6 +1,9 @@
[Contents](../Contents) \| [Previous (3.3 Error Checking)](03_Error_checking) \| [Next (3.5 Main Module)](05_Main_module)
# 3.4 Modules # 3.4 Modules
This section introduces the concept of modules. This section introduces the concept of modules and working with functions that span
multiple files.
### Modules and import ### Modules and import
@@ -27,9 +30,10 @@ b = foo.spam('Hello')
### Namespaces ### Namespaces
A module is a collection of named values and is sometimes said to be a *namespace*. A module is a collection of named values and is sometimes said to be a
The names are all of the global variables and functions defined in the source file. *namespace*. The names are all of the global variables and functions
After importing, the module name is used as a prefix. Hence the *namespace*. defined in the source file. After importing, the module name is used
as a prefix. Hence the *namespace*.
```python ```python
import foo import foo
@@ -39,12 +43,12 @@ b = foo.spam('Hello')
... ...
``` ```
The module name is tied to the file name (foo -> foo.py). The module name is directly tied to the file name (foo -> foo.py).
### Global Definitions ### Global Definitions
Everything defined in the *global* scope is what populates the module Everything defined in the *global* scope is what populates the module
namespace. `foo` in our previous example. Consider two modules namespace. Consider two modules
that define the same variable `x`. that define the same variable `x`.
```python ```python
@@ -118,14 +122,16 @@ def rectangular(r, theta):
return x, y return x, y
``` ```
It allows parts of a module to be used without having to type the module prefix. This allows parts of a module to be used without having to type the module prefix.
Useful for frequently used names. It's useful for frequently used names.
### Comments on importing ### Comments on importing
Variations on import do *not* change the way that modules work. Variations on import do *not* change the way that modules work.
```python ```python
import math
# vs
import math as m import math as m
# vs # vs
from math import cos, sin from math import cos, sin
@@ -135,7 +141,10 @@ from math import cos, sin
Specifically, `import` always executes the *entire* file and modules Specifically, `import` always executes the *entire* file and modules
are still isolated environments. are still isolated environments.
The `import module as` statement is only manipulating the names. The `import module as` statement is only changing the name locally.
The `from math import cos, sin` statement still loads the entire
math module behind the scenes. It's merely copying the `cos` and `sin`
names from the module into the local space after it's done.
### Module Loading ### Module Loading
@@ -151,6 +160,12 @@ Each module loads and executes only *once*.
>>> >>>
``` ```
**Caution:** A common confusion arises if you repeat an `import` statement after
changing the source code for a module. Because of the module cache `sys.modules`,
repeated imports always return the previously loaded module--even if a change
was made. The safest way to load modified code into Python is to quit and restart
the interpreter.
### Locating Modules ### Locating Modules
Python consults a path list (sys.path) when looking for modules. Python consults a path list (sys.path) when looking for modules.
@@ -166,12 +181,11 @@ Python consults a path list (sys.path) when looking for modules.
] ]
``` ```
Current working directory is usually first. The current working directory is usually first.
### Module Search Path ### Module Search Path
`sys.path` contains the search paths. As noted, `sys.path` contains the search paths.
You can manually adjust if you need to. You can manually adjust if you need to.
```python ```python
@@ -179,7 +193,7 @@ import sys
sys.path.append('/project/foo/pyfiles') sys.path.append('/project/foo/pyfiles')
``` ```
Paths are also added via environment variables. Paths can also be added via environment variables.
```python ```python
% env PYTHONPATH=/project/foo/pyfiles python3 % env PYTHONPATH=/project/foo/pyfiles python3
@@ -190,16 +204,26 @@ Python 3.6.0 (default, Feb 3 2017, 05:53:21)
['','/project/foo/pyfiles', ...] ['','/project/foo/pyfiles', ...]
``` ```
As a general rule, it should not be necessary to manually adjust
the module search path. However, it sometimes arises if you're
trying to import Python code that's in an unusual location or
not readily accessible from the current working directory.
## Exercises ## Exercises
For this exercise involving modules, it is critically important to For this exercise involving modules, it is critically important to
make sure you are running Python in a proper environment. Modules make sure you are running Python in a proper environment. Modules are
are usually when programmers encounter problems with the current working usually when programmers encounter problems with the current working
directory or with Python's path settings. directory or with Python's path settings. For this course, it is
assumed that you're writing all of your code in the `Work/` directory.
For best results, you should make sure you're also in that directory
when you launch the interpreter. If not, you need to make sure
`practical-python/Work` is added to `sys.path`.
### Exercise 3.11: Module imports ### Exercise 3.11: Module imports
In section 3, we created a general purpose function `parse_csv()` for parsing the contents of CSV datafiles. In section 3, we created a general purpose function `parse_csv()` for
parsing the contents of CSV datafiles.
Now, we’re going to see how to use that function in other programs. Now, we’re going to see how to use that function in other programs.
First, start in a new shell window. Navigate to the folder where you First, start in a new shell window. Navigate to the folder where you
@@ -217,7 +241,7 @@ Type "help", "copyright", "credits" or "license" for more information.
Once you’ve done that, try importing some of the programs you Once you’ve done that, try importing some of the programs you
previously wrote. You should see their output exactly as before. previously wrote. You should see their output exactly as before.
Just emphasize, importing a module runs its code. Just to emphasize, importing a module runs its code.
```python ```python
>>> import bounce >>> import bounce
@@ -272,9 +296,9 @@ Try importing a function so that you don’t need to include the module name:
In section 2, you wrote a program `report.py` that produced a stock report like this: In section 2, you wrote a program `report.py` that produced a stock report like this:
```shell ```
Name Shares Price Change Name Shares Price Change
---------- ---------- ---------- ---------- ---------- ---------- ---------- ----------
AA 100 39.91 7.71 AA 100 39.91 7.71
IBM 50 106.11 15.01 IBM 50 106.11 15.01
CAT 150 78.58 -4.86 CAT 150 78.58 -4.86
@@ -312,6 +336,6 @@ programs. `fileparse.py` which contains a general purpose
`parse_csv()` function. `report.py` which produces a nice report, but `parse_csv()` function. `report.py` which produces a nice report, but
also contains `read_portfolio()` and `read_prices()` functions. And also contains `read_portfolio()` and `read_prices()` functions. And
finally, `pcost.py` which computes the portfolio cost, but makes use finally, `pcost.py` which computes the portfolio cost, but makes use
of the code written for the `report.py` program. of the `read_portfolio()` function written for the `report.py` program.
[Contents](../Contents) \| [Previous (3.3 Error Checking)](03_Error_checking) \| [Next (3.5 Main Module)](05_Main_module) [Contents](../Contents) \| [Previous (3.3 Error Checking)](03_Error_checking) \| [Next (3.5 Main Module)](05_Main_module)

29
Notes/03_Program_organization/05_Main_module.md
View File

@@ -1,3 +1,5 @@
[Contents](../Contents) \| [Previous (3.4 Modules)](04_Modules) \| [Next (3.6 Design Discussion)](06_Design_discussion)
# 3.5 Main Module # 3.5 Main Module
This section introduces the concept of a main program or main module. This section introduces the concept of a main program or main module.
@@ -22,7 +24,7 @@ class myprog {
} }
``` ```
This is the first function that is being executing when an application is launched. This is the first function that executes when an application is launched.
### Python Main Module ### Python Main Module
@@ -34,11 +36,11 @@ bash % python3 prog.py
... ...
``` ```
Whatever module you give to the interpreter at startup becomes *main*. It doesn't matter the name. Whatever file you give to the interpreter at startup becomes *main*. It doesn't matter the name.
### `__main__` check ### `__main__` check
It is standard practice for modules that can run as a main script to use this convention: It is standard practice for modules that run as a main script to use this convention:
```python ```python
# prog.py # prog.py
@@ -53,22 +55,22 @@ Statements inclosed inside the `if` statement become the *main* program.
### Main programs vs. library imports ### Main programs vs. library imports
Any file can either run as main or as a library import: Any Python file can either run as main or as a library import:
```bash ```bash
bash % python3 prog.py # Running as main bash % python3 prog.py # Running as main
``` ```
```python ```python
import prog import prog # Running as library import
``` ```
In both cases, `__name__` is the name of the module. However, it will only be set to `__main__` if In both cases, `__name__` is the name of the module. However, it will only be set to `__main__` if
running as main. running as main.
As a general rule, you don't want statements that are part of the main Usually, you don't want statements that are part of the main program
program to execute on a library import. So, it's common to have an `if-`check in code to execute on a library import. So, it's common to have an `if-`check
that might be used either way. in code that might be used either way.
```python ```python
if __name__ == '__main__': if __name__ == '__main__':
@@ -221,7 +223,8 @@ bash % prog.py
### Script Template ### Script Template
Here is a common code template for Python programs that run as command-line scripts: Finally, here is a common code template for Python programs that run
as command-line scripts:
```python ```python
#!/usr/bin/env python3 #!/usr/bin/env python3
@@ -251,8 +254,9 @@ if __name__ == '__main__':
### Exercise 3.15: `main()` functions ### Exercise 3.15: `main()` functions
In the file `report.py` add a `main()` function that accepts a list of command line options and produces the same output as before. In the file `report.py` add a `main()` function that accepts a list of
You should be able to run it interatively like this: command line options and produces the same output as before. You
should be able to run it interatively like this:
```python ```python
>>> import report >>> import report
@@ -280,7 +284,8 @@ Total cost: 44671.15
### Exercise 3.16: Making Scripts ### Exercise 3.16: Making Scripts
Modify the `report.py` and `pcost.py` programs so that they can execute as a script on the command line: Modify the `report.py` and `pcost.py` programs so that they can
execute as a script on the command line:
```bash ```bash
bash $ python3 report.py Data/portfolio.csv Data/prices.csv bash $ python3 report.py Data/portfolio.csv Data/prices.csv

27
Notes/03_Program_organization/06_Design_discussion.md
View File

@@ -1,6 +1,8 @@
[Contents](../Contents) \| [Previous (3.5 Main module)](05_Main_module) \| [Next (4 Classes)](../04_Classes_objects/00_Overview)
# 3.6 Design Discussion # 3.6 Design Discussion
In this section we consider some design decisions made in code so far. In this section we reconsider a design decision made earlier.
### Filenames versus Iterables ### Filenames versus Iterables
@@ -35,15 +37,17 @@ with open('file.csv') as f:
* Which of these functions do you prefer? Why? * Which of these functions do you prefer? Why?
* Which of these functions is more flexible? * Which of these functions is more flexible?
### Deep Idea: "Duck Typing" ### Deep Idea: "Duck Typing"
[Duck Typing](https://en.wikipedia.org/wiki/Duck_typing) is a computer programming concept to determine whether an object can be used for a particular purpose. It is an application of the [duck test](https://en.wikipedia.org/wiki/Duck_test). [Duck Typing](https://en.wikipedia.org/wiki/Duck_typing) is a computer
programming concept to determine whether an object can be used for a
particular purpose. It is an application of the [duck
test](https://en.wikipedia.org/wiki/Duck_test).
> If it looks like a duck, swims like a duck, and quacks like a duck, then it probably is a duck. > If it looks like a duck, swims like a duck, and quacks like a duck, then it probably is a duck.
In our previous example that reads the lines, our `read_data` expects In the second version of `read_data()` above, the function expects any
any iterable object. Not just the lines of a file. iterable object. Not just the lines of a file.
```python ```python
def read_data(lines): def read_data(lines):
@@ -76,7 +80,7 @@ data = read_data(lines)
There is considerable flexibility with this design. There is considerable flexibility with this design.
*Question: Shall we embrace or fight this flexibility?* *Question: Should we embrace or fight this flexibility?*
### Library Design Best Practices ### Library Design Best Practices
@@ -87,10 +91,10 @@ Don't restrict your options. With great flexibility comes great power.
### Exercise 3.17: From filenames to file-like objects ### Exercise 3.17: From filenames to file-like objects
In this section, you worked on a file `fileparse.py` that contained a You've now created a file `fileparse.py` that contained a
function `parse_csv()`. The function worked like this: function `parse_csv()`. The function worked like this:
```pycon ```python
>>> import fileparse >>> import fileparse
>>> portfolio = fileparse.parse_csv('Data/portfolio.csv', types=[str,int,float]) >>> portfolio = fileparse.parse_csv('Data/portfolio.csv', types=[str,int,float])
>>> >>>
@@ -103,8 +107,8 @@ with any file-like/iterable object. For example:
``` ```
>>> import fileparse >>> import fileparse
>>> import gzip >>> import gzip
>>> with gzip.open('Data/portfolio.csv.gz', 'rt') as f: >>> with gzip.open('Data/portfolio.csv.gz', 'rt') as file:
... port = fileparse.parse_csv(f, types=[str,int,float]) ... port = fileparse.parse_csv(file, types=[str,int,float])
... ...
>>> lines = ['name,shares,price', 'AA,34.23,100', 'IBM,50,91.1', 'HPE,75,45.1'] >>> lines = ['name,shares,price', 'AA,34.23,100', 'IBM,50,91.1', 'HPE,75,45.1']
>>> port = fileparse.parse_csv(lines, types=[str,int,float]) >>> port = fileparse.parse_csv(lines, types=[str,int,float])
@@ -120,8 +124,7 @@ In this new code, what happens if you pass a filename as before?
>>> >>>
``` ```
With flexibility comes power and with power comes responsibility. Sometimes you'll Yes, you'll need to be careful. Could you add a safety check to avoid this?
need to be careful.
### Exercise 3.18: Fixing existing functions ### Exercise 3.18: Fixing existing functions