diff --git a/Notes/03_Program_organization/01_Script.md b/Notes/03_Program_organization/01_Script.md index 8c8ab31..073d0d8 100644 --- a/Notes/03_Program_organization/01_Script.md +++ b/Notes/03_Program_organization/01_Script.md @@ -1,3 +1,5 @@ +[Contents](../Contents) \| [Previous (2.7 Object Model)](../02_Working_with_data/07_Objects) \| [Next (3.2 More on Functions)](02_More_functions) + # 3.1 Scripting In this part we look more closely at the practice of writing Python @@ -16,7 +18,7 @@ statement3 ... ``` -We have been writing scripts to this point. +We have mostly been writing scripts to this point. ### A Problem @@ -28,7 +30,7 @@ organized. ### Defining Things -You must always define things before they get used later on in a program. +Names must always be defined before they get used later. ```python def square(x): @@ -41,11 +43,12 @@ z = square(b) # Requires `square` and `b` to be defined ``` **The order is important.** -You almost always put the definitions of variables an functions near the beginning. +You almost always put the definitions of variables and functions near the top. ### Defining Functions It is a good idea to put all of the code related to a single *task* all in one place. +Use a function. ```python def read_prices(filename): @@ -85,7 +88,7 @@ def foo(): help(math) ``` -There are no *special* statements in Python. +There are no *special* statements in Python (which makes it easy to remember). ### Function Definition @@ -106,13 +109,14 @@ def foo(x): bar(x) ``` -Functions must only be defined before they are actually *used* (or called) during program execution. +Functions must only be defined prior to actually being *used* (or called) during program execution. ```python foo(3) # foo must be defined already ``` -Stylistically, it is probably more common to see functions defined in a *bottom-up* fashion. +Stylistically, it is probably more common to see functions defined in +a *bottom-up* fashion. ### Bottom-up Style @@ -137,24 +141,26 @@ def spam(x): spam(42) # Code that uses the functions appears at the end ``` -Later functions build upon earlier functions. +Later functions build upon earlier functions. Again, this is only +a point of style. The only thing that matters in the above program +is that the call to `spam(42)` go last. ### Function Design Ideally, functions should be a *black box*. They should only operate on passed inputs and avoid global variables -and mysterious side-effects. Main goals: *Modularity* and *Predictability*. +and mysterious side-effects. Your main goals: *Modularity* and *Predictability*. ### Doc Strings -A good practice is to include documentations in the form of -doc-strings. Doc-strings are strings written immediately after the +It's good practice to include documentation in the form of a +doc-string. Doc-strings are strings written immediately after the name of the function. They feed `help()`, IDEs and other tools. ```python def read_prices(filename): ''' - Read prices from a CSV file of name,price + Read prices from a CSV file of name,price data ''' prices = {} with open(filename) as f: @@ -164,14 +170,19 @@ def read_prices(filename): return prices ``` +A good practice for doc strings is to write a short one sentence +summary of what the function does. If more information is needed, +include a short example of usage along with a more detailed +description of the arguments. + ### Type Annotations -You can also add some optional type annotations to your function definitions. +You can also add optional type hints to function definitions. ```python def read_prices(filename: str) -> dict: ''' - Read prices from a CSV file of name,price + Read prices from a CSV file of name,price data ''' prices = {} with open(filename) as f: @@ -181,13 +192,15 @@ def read_prices(filename: str) -> dict: return prices ``` -These do nothing. It is purely informational. -They may be used by IDEs, code checkers, etc. +The hints do nothing operationally. They are purely informational. +However, they may be used by IDEs, code checkers, and other tools +to do more. ## Exercises -In section 2, you wrote a program called `report.py` that printed out a report showing the performance of a stock portfolio. -This program consisted of some functions. For example: +In section 2, you wrote a program called `report.py` that printed out +a report showing the performance of a stock portfolio. This program +consisted of some functions. For example: ```python # report.py @@ -215,8 +228,9 @@ def read_portfolio(filename): ... ``` -However, there were also portions of the program that just performed a series of scripted calculations. -This code appeared near the end of the program. For example: +However, there were also portions of the program that just performed a +series of scripted calculations. This code appeared near the end of +the program. For example: ```python ... @@ -231,7 +245,8 @@ for row in report: ... ``` -In this exercise, we’re going take this program and organize it a little more strongly around the use of functions. +In this exercise, we’re going take this program and organize it a +little more strongly around the use of functions. ### Exercise 3.1: Structuring a program as a collection of functions @@ -242,10 +257,12 @@ functions. Specifically: * Create a function `print_report(report)` that prints out the report. * Change the last part of the program so that it is nothing more than a series of function calls and no other computation. -### Exercise 3.2: Creating a function for program execution +### Exercise 3.2: Creating a top-level function for program execution -Take the last part of your program and package it into a single function `portfolio_report(portfolio_filename, prices_filename)`. -Have the function work so that the following function call creates the report as before: +Take the last part of your program and package it into a single +function `portfolio_report(portfolio_filename, prices_filename)`. +Have the function work so that the following function call creates the +report as before: ```python portfolio_report('Data/portfolio.csv', 'Data/prices.csv') @@ -256,8 +273,9 @@ of function definitions followed by a single function call to `portfolio_report()` at the very end (which executes all of the steps involved in the program). -By turning your program into a single function, it becomes easy to run it on different inputs. -For example, try these statements interactively after running your program: +By turning your program into a single function, it becomes easy to run +it on different inputs. For example, try these statements +interactively after running your program: ```python >>> portfolio_report('Data/portfolio2.csv', 'Data/prices.csv') @@ -272,4 +290,13 @@ For example, try these statements interactively after running your program: >>> ``` +### Commentary + +Python makes it very easy to write relatively unstructured scripting code +where you just have a file with a sequence of statements in it. In the +big picture, it's almost always better to utilize functions whenever +you can. At some point, that script is going to grow and you'll wish +you had a bit more organization. Also, a little known fact is that Python +runs a bit faster if you use functions. + [Contents](../Contents) \| [Previous (2.7 Object Model)](../02_Working_with_data/07_Objects) \| [Next (3.2 More on Functions)](02_More_functions) diff --git a/Notes/03_Program_organization/02_More_functions.md b/Notes/03_Program_organization/02_More_functions.md index 0b7c1f9..8552633 100644 --- a/Notes/03_Program_organization/02_More_functions.md +++ b/Notes/03_Program_organization/02_More_functions.md @@ -1,6 +1,10 @@ +[Contents](../Contents) \| [Previous (3.1 Scripting)](01_Script) \| [Next (3.3 Error Checking)](03_Error_checking) + # 3.2 More on Functions -This section fills in a few more details about how functions work and are defined. +Although functions were introduced earlier, very few details were provided on how +they actually work at a deeper level. This section aims to fill in some gaps +and discuss matters such as calling conventions, scoping rules, and more. ### Calling a Function @@ -25,7 +29,8 @@ prices = read_prices(filename='prices.csv', debug=True) ### Default Arguments -Sometimes you want an optional argument. +Sometimes you want an argument to be optional. If so, assign a default value +in the function definition. ```python def read_prices(filename, debug=False): @@ -53,7 +58,8 @@ parse_data(data, debug=True) parse_data(data, debug=True, ignore_errors=True) ``` -Keyword arguments improve code clarity. +In most cases, keyword arguments improve code clarity--especially for arguments that +serve as flags or which are related to optional features. ### Design Best Practices @@ -67,7 +73,7 @@ d = read_prices('prices.csv', debug=True) Python development tools will show the names in help features and documentation. -### Return Values +### Returning Values The `return` statement returns a value @@ -76,7 +82,7 @@ def square(x): return x * x ``` -If no return value or `return` not specified, `None` is returned. +If no return value is given or `return` is missing, `None` is returned. ```python def bar(x): @@ -94,8 +100,8 @@ b = foo(4) # b = None ### Multiple Return Values -Functions can only return one value. -However, a function may return multiple values by returning a tuple. +Functions can only return one value. However, a function may return +multiple values by returning them in a tuple. ```python def divide(a,b): @@ -124,18 +130,19 @@ def foo(): ``` Variables assignments occur outside and inside function definitions. -Variables defined outside are "global". Variables inside a function are "local". +Variables defined outside are "global". Variables inside a function +are "local". ### Local Variables -Variables inside functions are private. +Variables assigned inside functions are private. ```python def read_portfolio(filename): portfolio = [] for line in open(filename): - fields = line.split() - s = (fields[0],int(fields[1]),float(fields[2])) + fields = line.split(',') + s = (fields[0], int(fields[1]), float(fields[2])) portfolio.append(s) return portfolio ``` @@ -143,8 +150,8 @@ def read_portfolio(filename): In this example, `filename`, `portfolio`, `line`, `fields` and `s` are local variables. Those variables are not retained or accessible after the function call. -```pycon ->>> stocks = read_portfolio('stocks.dat') +```python +>>> stocks = read_portfolio('portfolio.csv') >>> fields Traceback (most recent call last): File "", line 1, in ? @@ -152,11 +159,12 @@ NameError: name 'fields' is not defined >>> ``` -They also can't conflict with variables found elsewhere. +Locals also can't conflict with variables found elsewhere. ### Global Variables -Functions can freely access the values of globals. +Functions can freely access the values of globals defined in the same +file. ```python name = 'Dave' @@ -185,20 +193,24 @@ If you must modify a global variable you must declare it as such. ```python name = 'Dave' + def spam(): global name name = 'Guido' # Changes the global name above ``` -The global declaration must appear before its use. Having seen this, -know that it is considered poor form. In fact, try to avoid entirely +The global declaration must appear before its use and the corresponding +variable must exist in the same file as the function. Having seen this, +know that it is considered poor form. In fact, try to avoid `global` entirely if you can. If you need a function to modify some kind of state outside of the function, it's better to use a class instead (more on this later). ### Argument Passing -When you call a function, the argument variables are names for passed values. -If mutable data types are passed (e.g. lists, dicts), they can be modified *in-place*. +When you call a function, the argument variables are names that refer +to the passed values. These values are NOT copies (see [section +2.7](../02_Working_with_data/07_Objects)). If mutable data types are +passed (e.g. lists, dicts), they can be modified *in-place*. ```python def foo(items): @@ -213,7 +225,8 @@ print(a) # [1, 2, 3, 42] ### Reassignment vs Modifying -Make sure you understand the subtle difference between modifying a value and reassigning a variable name. +Make sure you understand the subtle difference between modifying a +value and reassigning a variable name. ```python def foo(items): @@ -225,19 +238,22 @@ print(a) # [1, 2, 3, 42] # VS def bar(items): - items = [4,5,6] # Reassigns `items` variable + items = [4,5,6] # Changes local `items` variable to point to a different object b = [1, 2, 3] bar(b) print(b) # [1, 2, 3] ``` -*Reminder: Variable assignment never overwrites memory. The name is simply bound to a new value.* +*Reminder: Variable assignment never overwrites memory. The name is merely bound to a new value.* ## Exercises -This exercise involves a lot of steps and putting concepts together from past exercises. -The final solution is only about 25 lines of code, but take your time and make sure you understand each part. +This set of exercises have you implement what is, perhaps, the most +powerful and difficult part of the course. There are a lot of steps +and many concepts from past exercises are put together all at once. +The final solution is only about 25 lines of code, but take your time +and make sure you understand each part. A central part of your `report.py` program focuses on the reading of CSV files. For example, the function `read_portfolio()` reads a file @@ -251,12 +267,13 @@ If you were doing a lot of file parsing for real, you’d probably want to clean some of this up and make it more general purpose. That's our goal. -Start this exercise by creating a new file called `fileparse.py`. This is where we will be doing our work. +Start this exercise by creating a new file called +`Work/fileparse.py`. This is where we will be doing our work. ### Exercise 3.3: Reading CSV Files To start, let’s just focus on the problem of reading a CSV file into a -list of dictionaries. In the file `fileparse.py`, define a simple +list of dictionaries. In the file `fileparse.py`, define a function that looks like this: ```python @@ -290,20 +307,23 @@ Try it out: Hint: `python3 -i fileparse.py`. -```pycon +```python >>> portfolio = parse_csv('Data/portfolio.csv') >>> portfolio [{'price': '32.20', 'name': 'AA', 'shares': '100'}, {'price': '91.10', 'name': 'IBM', 'shares': '50'}, {'price': '83.44', 'name': 'CAT', 'shares': '150'}, {'price': '51.23', 'name': 'MSFT', 'shares': '200'}, {'price': '40.37', 'name': 'GE', 'shares': '95'}, {'price': '65.10', 'name': 'MSFT', 'shares': '50'}, {'price': '70.44', 'name': 'IBM', 'shares': '100'}] >>> ``` -This is great except that you can’t do any kind of useful calculation with the data because everything is represented as a string. -We’ll fix this shortly, but let’s keep building on it. +This is good except that you can’t do any kind of useful calculation +with the data because everything is represented as a string. We’ll +fix this shortly, but let’s keep building on it. ### Exercise 3.4: Building a Column Selector -In many cases, you’re only interested in selected columns from a CSV file, not all of the data. -Modify the `parse_csv()` function so that it optionally allows user-specified columns to be picked out as follows: +In many cases, you’re only interested in selected columns from a CSV +file, not all of the data. Modify the `parse_csv()` function so that +it optionally allows user-specified columns to be picked out as +follows: ```python >>> # Read all of the data @@ -311,14 +331,14 @@ Modify the `parse_csv()` function so that it optionally allows user-specified co >>> portfolio [{'price': '32.20', 'name': 'AA', 'shares': '100'}, {'price': '91.10', 'name': 'IBM', 'shares': '50'}, {'price': '83.44', 'name': 'CAT', 'shares': '150'}, {'price': '51.23', 'name': 'MSFT', 'shares': '200'}, {'price': '40.37', 'name': 'GE', 'shares': '95'}, {'price': '65.10', 'name': 'MSFT', 'shares': '50'}, {'price': '70.44', 'name': 'IBM', 'shares': '100'}] ->>> # Read some of the data +>>> # Read only some of the data >>> shares_held = parse_csv('portfolio.csv', select=['name','shares']) >>> shares_held [{'name': 'AA', 'shares': '100'}, {'name': 'IBM', 'shares': '50'}, {'name': 'CAT', 'shares': '150'}, {'name': 'MSFT', 'shares': '200'}, {'name': 'GE', 'shares': '95'}, {'name': 'MSFT', 'shares': '50'}, {'name': 'IBM', 'shares': '100'}] >>> ``` -An example of a column selector was given in [Exercise 2.23](../02_Working_with_data/06_List_comprehension). +An example of a column selector was given in [Exercise 2.23](../02_Working_with_data/06_List_comprehension). However, here’s one way to do it: ```python @@ -358,17 +378,18 @@ def parse_csv(filename, select=None): return records ``` -There are a number of tricky bits to this part. Probably the most important one is the mapping of the column selections to row indices. +There are a number of tricky bits to this part. Probably the most +important one is the mapping of the column selections to row indices. For example, suppose the input file had the following headers: -```pycon +```python >>> headers = ['name', 'date', 'time', 'shares', 'price'] >>> ``` Now, suppose the selected columns were as follows: -```pycon +```python >>> select = ['name', 'shares'] >>> ``` @@ -376,7 +397,7 @@ Now, suppose the selected columns were as follows: To perform the proper selection, you have to map the selected column names to column indices in the file. That’s what this step is doing: -```pycon +```python >>> indices = [headers.index(colname) for colname in select ] >>> indices [0, 3] @@ -386,7 +407,7 @@ That’s what this step is doing: In other words, "name" is column 0 and "shares" is column 3. When you read a row of data from the file, the indices are used to filter it: -```pycon +```python >>> row = ['AA', '6/11/2007', '9:50am', '100', '32.20' ] >>> row = [ row[index] for index in indices ] >>> row @@ -396,10 +417,10 @@ When you read a row of data from the file, the indices are used to filter it: ### Exercise 3.5: Performing Type Conversion -Modify the `parse_csv()` function so that it optionally allows type-conversions to be applied to the returned data. -For example: +Modify the `parse_csv()` function so that it optionally allows +type-conversions to be applied to the returned data. For example: -```pycon +```python >>> portfolio = parse_csv('Data/portfolio.csv', types=[str, int, float]) >>> portfolio [{'price': 32.2, 'name': 'AA', 'shares': 100}, {'price': 91.1, 'name': 'IBM', 'shares': 50}, {'price': 83.44, 'name': 'CAT', 'shares': 150}, {'price': 51.23, 'name': 'MSFT', 'shares': 200}, {'price': 40.37, 'name': 'GE', 'shares': 95}, {'price': 65.1, 'name': 'MSFT', 'shares': 50}, {'price': 70.44, 'name': 'IBM', 'shares': 100}] @@ -410,8 +431,8 @@ For example: >>> ``` -You already explored this in [Exercise 2.24](../02_Working_with_data/07_Objects). You'll need to insert the -following fragment of code into your solution: +You already explored this in [Exercise 2.24](../02_Working_with_data/07_Objects). +You'll need to insert the following fragment of code into your solution: ```python ... @@ -420,7 +441,7 @@ if types: ... ``` -### Exercise 3.6: Working with Headers +### Exercise 3.6: Working without Headers Some CSV files don’t include any header information. For example, the file `prices.csv` looks like this: @@ -433,8 +454,8 @@ For example, the file `prices.csv` looks like this: ... ``` -Modify the `parse_csv()` function so that it can work with such files by creating a list of tuples instead. -For example: +Modify the `parse_csv()` function so that it can work with such files +by creating a list of tuples instead. For example: ```python >>> prices = parse_csv('Data/prices.csv', types=[str,float], has_headers=False) @@ -450,8 +471,10 @@ column names to use for keys. ### Exercise 3.7: Picking a different column delimitier -Although CSV files are pretty common, it’s also possible that you could encounter a file that uses a different column separator such as a tab or space. -For example, the file `Data/portfolio.dat` looks like this: +Although CSV files are pretty common, it’s also possible that you +could encounter a file that uses a different column separator such as +a tab or space. For example, the file `Data/portfolio.dat` looks like +this: ```csv name shares price @@ -464,26 +487,30 @@ name shares price "IBM" 100 70.44 ``` -The `csv.reader()` function allows a different delimiter to be given as follows: +The `csv.reader()` function allows a different column delimiter to be given as follows: ```python rows = csv.reader(f, delimiter=' ') ``` -Modify your `parse_csv()` function so that it also allows the delimiter to be changed. +Modify your `parse_csv()` function so that it also allows the +delimiter to be changed. For example: -```pycon +```python >>> portfolio = parse_csv('Data/portfolio.dat', types=[str, int, float], delimiter=' ') >>> portfolio [{'price': '32.20', 'name': 'AA', 'shares': '100'}, {'price': '91.10', 'name': 'IBM', 'shares': '50'}, {'price': '83.44', 'name': 'CAT', 'shares': '150'}, {'price': '51.23', 'name': 'MSFT', 'shares': '200'}, {'price': '40.37', 'name': 'GE', 'shares': '95'}, {'price': '65.10', 'name': 'MSFT', 'shares': '50'}, {'price': '70.44', 'name': 'IBM', 'shares': '100'}] >>> ``` -If you’ve made it this far, you’ve created a nice library function that’s genuinely useful. -You can use it to parse arbitrary CSV files, select out columns of -interest, perform type conversions, without having to worry too much -about the inner workings of files or the `csv` module. +### Commentary + +If you’ve made it this far, you’ve created a nice library function +that’s genuinely useful. You can use it to parse arbitrary CSV files, +select out columns of interest, perform type conversions, without +having to worry too much about the inner workings of files or the +`csv` module. [Contents](../Contents) \| [Previous (3.1 Scripting)](01_Script) \| [Next (3.3 Error Checking)](03_Error_checking) diff --git a/Notes/03_Program_organization/03_Error_checking.md b/Notes/03_Program_organization/03_Error_checking.md index e2e2f13..c8c574a 100644 --- a/Notes/03_Program_organization/03_Error_checking.md +++ b/Notes/03_Program_organization/03_Error_checking.md @@ -1,11 +1,15 @@ +[Contents](../Contents) \| [Previous (3.2 More on Functions)](02_More_functions) \| [Next (3.4 Modules)](04_Modules) + # 3.3 Error Checking -This section discusses some aspects of error checking and exception handling. +Although exceptions were introduced earlier, this section fills in some additional +details about error checking and exception handling. ### How programs fail -Python performs no checking or validation of function argument types or values. -A function will work on any data that is compatible with the statements in the function. +Python performs no checking or validation of function argument types +or values. A function will work on any data that is compatible with +the statements in the function. ```python def add(x, y): @@ -16,7 +20,7 @@ add('Hello', 'World') # 'HelloWorld' add('3', '4') # '34' ``` -If there are errors in a function, they will show up at run time (as an exception). +If there are errors in a function, they appear at run time (as an exception). ```python def add(x, y): @@ -38,8 +42,8 @@ Exceptions are used to signal errors. To raise an exception yourself, use `raise` statement. ```python -if name not in names: - raise RuntimeError('Name not found') +if name not in authorized: + raise RuntimeError(f'{name} not authorized') ``` To catch an exception use `try-except`. @@ -78,7 +82,8 @@ def foo(): foo() ``` -To handle the exception, use the `except` block. You can add any statements you want to handle the error. +To handle the exception, put statements in the `except` block. You can add any +statements you want to handle the error. ```python def grok(): ... @@ -95,7 +100,8 @@ def bar(): bar() ``` -After handling, execution resumes with the first statement after the `try-except`. +After handling, execution resumes with the first statement after the +`try-except`. ```python def grok(): ... @@ -117,8 +123,10 @@ bar() ### Built-in Exceptions -There are about two-dozen built-in exceptions. -This is not an exhaustive list. Check the documentation for more. +There are about two-dozen built-in exceptions. Usually the name of +the exception is indicative of what's wrong (e.g., a `ValueError` is +raised because you supplied a bad value). This is not an +exhaustive list. Check the documentation for more. ```python ArithmeticError @@ -141,22 +149,23 @@ ValueError ### Exception Values -Most exceptions have an associated value. It contains more information about what's wrong. +Exceptions have an associated value. It contains more specific +information about what's wrong. ```python raise RuntimeError('Invalid user name') ``` -This value is passed to the variable supplied in `except`. +This value is part of the exception instance that's placed in the variable supplied to `except`. ```python try: ... -except RuntimeError as e: # `e` holds the value raised +except RuntimeError as e: # `e` holds the exception raised ... ``` -The value is an instance of the exception type. However, it often looks like a string when +`e` is an instance of the exception type. However, it often looks like a string when printed. ```python @@ -166,7 +175,7 @@ except RuntimeError as e: ### Catching Multiple Errors -You can catch different kinds of exceptions with multiple `except` blocks. +You can catch different kinds of exceptions using multiple `except` blocks. ```python try: @@ -181,7 +190,7 @@ except KeyboardInterrupt as e: ... ``` -Alternatively, if the block to handle them is the same, you can group them: +Alternatively, if the statements to handle them is the same, you can group them: ```python try: @@ -197,12 +206,12 @@ To catch any exception, use `Exception` like this: ```python try: ... -except Exception: +except Exception: # DANGER. See below print('An error occurred') ``` -In general, writing code like that is a bad idea because you'll have no idea -why it failed. +In general, writing code like that is a bad idea because you'll have +no idea why it failed. ### Wrong Way to Catch Errors @@ -215,13 +224,13 @@ except Exception: print('Computer says no') ``` -This swallows all possible errors. It may make it impossible to debug +This catches all possible errors and it may make it impossible to debug when the code is failing for some reason you didn't expect at all (e.g. uninstalled Python module, etc.). ### Somewhat Better Approach -This is a more sane approach. +If you're going to catch all errors, this is a more sane approach. ```python try: @@ -234,9 +243,9 @@ It reports a specific reason for failure. It is almost always a good idea to have some mechanism for viewing/reporting errors when you write code that catches all possible exceptions. -In general though, it's better to catch the error more narrowly. Only -catch the errors you can actually deal with. Let other errors pass to -other code. +In general though, it's better to catch the error as narrowly as is +reasonable. Only catch the errors you can actually handle. Let +other errors pass by--maybe some other code can handle them. ### Reraising an Exception @@ -250,7 +259,8 @@ except Exception as e: raise ``` -It allows you to take action (e.g. logging) and pass the error on to the caller. +This allows you to take action (e.g. logging) and pass the error on to +the caller. ### Exception Best Practices @@ -261,7 +271,8 @@ and sanely keep going. ### `finally` statement -It specifies code that must fun regardless of whether or not an exception occurs. +It specifies code that must run regardless of whether or not an +exception occurs. ```python lock = Lock() @@ -273,11 +284,11 @@ finally: lock.release() # this will ALWAYS be executed. With and without exception. ``` -Comonly used to properly manage resources (especially locks, files, etc.). +Commonly used to safely manage resources (especially locks, files, etc.). ### `with` statement -In modern code, `try-finally` often replaced with the `with` statement. +In modern code, `try-finally` is often replaced with the `with` statement. ```python lock = Lock() @@ -296,8 +307,9 @@ with open(filename) as f: # File closed ``` -It defines a usage *context* for a resource. When execution leaves that context, -resources are released. `with` only works with certain objects. +`with` defines a usage *context* for a resource. When execution +leaves that context, resources are released. `with` only works with +certain objects that have been specifically programmed to support it. ## Exercises @@ -308,8 +320,7 @@ user-specified columns to be selected, but that only works if the input data file has column headers. Modify the code so that an exception gets raised if both the `select` -and `has_headers=False` arguments are passed. -For example: +and `has_headers=False` arguments are passed. For example: ```python >>> parse_csv('Data/prices.csv', select=['name','price'], has_headers=False) @@ -335,6 +346,7 @@ in a non-sensical mode (e.g., using a feature that requires column headers, but simultaneously specifying that there are no headers). This indicates a programming error on the part of the calling code. +Checking for cases that "aren't supposed to happen" is often a good idea. ### Exercise 3.9: Catching exceptions @@ -357,9 +369,9 @@ Modify the `parse_csv()` function to catch all `ValueError` exceptions generated during record creation and print a warning message for rows that can’t be converted. -The message should include the row number and information about the reason why it failed. -To test your function, try reading the file `Data/missing.csv` above. -For example: +The message should include the row number and information about the +reason why it failed. To test your function, try reading the file +`Data/missing.csv` above. For example: ```python >>> portfolio = parse_csv('Data/missing.csv', types=[str, int, float]) @@ -375,8 +387,8 @@ Row 7: Reason invalid literal for int() with base 10: '' ### Exercise 3.10: Silencing Errors -Modify the `parse_csv()` function so that parsing error messages can be silenced if explicitly desired by the user. -For example: +Modify the `parse_csv()` function so that parsing error messages can +be silenced if explicitly desired by the user. For example: ```python >>> portfolio = parse_csv('Data/missing.csv', types=[str,int,float], silence_errors=True) diff --git a/Notes/03_Program_organization/04_Modules.md b/Notes/03_Program_organization/04_Modules.md index 9d6601d..3dcd21d 100644 --- a/Notes/03_Program_organization/04_Modules.md +++ b/Notes/03_Program_organization/04_Modules.md @@ -1,6 +1,9 @@ +[Contents](../Contents) \| [Previous (3.3 Error Checking)](03_Error_checking) \| [Next (3.5 Main Module)](05_Main_module) + # 3.4 Modules -This section introduces the concept of modules. +This section introduces the concept of modules and working with functions that span +multiple files. ### Modules and import @@ -27,9 +30,10 @@ b = foo.spam('Hello') ### Namespaces -A module is a collection of named values and is sometimes said to be a *namespace*. -The names are all of the global variables and functions defined in the source file. -After importing, the module name is used as a prefix. Hence the *namespace*. +A module is a collection of named values and is sometimes said to be a +*namespace*. The names are all of the global variables and functions +defined in the source file. After importing, the module name is used +as a prefix. Hence the *namespace*. ```python import foo @@ -39,12 +43,12 @@ b = foo.spam('Hello') ... ``` -The module name is tied to the file name (foo -> foo.py). +The module name is directly tied to the file name (foo -> foo.py). ### Global Definitions Everything defined in the *global* scope is what populates the module -namespace. `foo` in our previous example. Consider two modules +namespace. Consider two modules that define the same variable `x`. ```python @@ -118,14 +122,16 @@ def rectangular(r, theta): return x, y ``` -It allows parts of a module to be used without having to type the module prefix. -Useful for frequently used names. +This allows parts of a module to be used without having to type the module prefix. +It's useful for frequently used names. ### Comments on importing Variations on import do *not* change the way that modules work. ```python +import math +# vs import math as m # vs from math import cos, sin @@ -135,7 +141,10 @@ from math import cos, sin Specifically, `import` always executes the *entire* file and modules are still isolated environments. -The `import module as` statement is only manipulating the names. +The `import module as` statement is only changing the name locally. +The `from math import cos, sin` statement still loads the entire +math module behind the scenes. It's merely copying the `cos` and `sin` +names from the module into the local space after it's done. ### Module Loading @@ -151,6 +160,12 @@ Each module loads and executes only *once*. >>> ``` +**Caution:** A common confusion arises if you repeat an `import` statement after +changing the source code for a module. Because of the module cache `sys.modules`, +repeated imports always return the previously loaded module--even if a change +was made. The safest way to load modified code into Python is to quit and restart +the interpreter. + ### Locating Modules Python consults a path list (sys.path) when looking for modules. @@ -166,12 +181,11 @@ Python consults a path list (sys.path) when looking for modules. ] ``` -Current working directory is usually first. +The current working directory is usually first. ### Module Search Path -`sys.path` contains the search paths. - +As noted, `sys.path` contains the search paths. You can manually adjust if you need to. ```python @@ -179,7 +193,7 @@ import sys sys.path.append('/project/foo/pyfiles') ``` -Paths are also added via environment variables. +Paths can also be added via environment variables. ```python % env PYTHONPATH=/project/foo/pyfiles python3 @@ -190,16 +204,26 @@ Python 3.6.0 (default, Feb 3 2017, 05:53:21) ['','/project/foo/pyfiles', ...] ``` +As a general rule, it should not be necessary to manually adjust +the module search path. However, it sometimes arises if you're +trying to import Python code that's in an unusual location or +not readily accessible from the current working directory. + ## Exercises For this exercise involving modules, it is critically important to -make sure you are running Python in a proper environment. Modules -are usually when programmers encounter problems with the current working -directory or with Python's path settings. +make sure you are running Python in a proper environment. Modules are +usually when programmers encounter problems with the current working +directory or with Python's path settings. For this course, it is +assumed that you're writing all of your code in the `Work/` directory. +For best results, you should make sure you're also in that directory +when you launch the interpreter. If not, you need to make sure +`practical-python/Work` is added to `sys.path`. ### Exercise 3.11: Module imports -In section 3, we created a general purpose function `parse_csv()` for parsing the contents of CSV datafiles. +In section 3, we created a general purpose function `parse_csv()` for +parsing the contents of CSV datafiles. Now, we’re going to see how to use that function in other programs. First, start in a new shell window. Navigate to the folder where you @@ -217,7 +241,7 @@ Type "help", "copyright", "credits" or "license" for more information. Once you’ve done that, try importing some of the programs you previously wrote. You should see their output exactly as before. -Just emphasize, importing a module runs its code. +Just to emphasize, importing a module runs its code. ```python >>> import bounce @@ -272,16 +296,16 @@ Try importing a function so that you don’t need to include the module name: In section 2, you wrote a program `report.py` that produced a stock report like this: -```shell - Name Shares Price Change - ---------- ---------- ---------- ---------- - AA 100 39.91 7.71 - IBM 50 106.11 15.01 - CAT 150 78.58 -4.86 - MSFT 200 30.47 -20.76 - GE 95 37.38 -2.99 - MSFT 50 30.47 -34.63 - IBM 100 106.11 35.67 +``` + Name Shares Price Change +---------- ---------- ---------- ---------- + AA 100 39.91 7.71 + IBM 50 106.11 15.01 + CAT 150 78.58 -4.86 + MSFT 200 30.47 -20.76 + GE 95 37.38 -2.99 + MSFT 50 30.47 -34.63 + IBM 100 106.11 35.67 ``` Take that program and modify it so that all of the input file @@ -312,6 +336,6 @@ programs. `fileparse.py` which contains a general purpose `parse_csv()` function. `report.py` which produces a nice report, but also contains `read_portfolio()` and `read_prices()` functions. And finally, `pcost.py` which computes the portfolio cost, but makes use -of the code written for the `report.py` program. +of the `read_portfolio()` function written for the `report.py` program. [Contents](../Contents) \| [Previous (3.3 Error Checking)](03_Error_checking) \| [Next (3.5 Main Module)](05_Main_module) diff --git a/Notes/03_Program_organization/05_Main_module.md b/Notes/03_Program_organization/05_Main_module.md index 9e52789..6e5721a 100644 --- a/Notes/03_Program_organization/05_Main_module.md +++ b/Notes/03_Program_organization/05_Main_module.md @@ -1,3 +1,5 @@ +[Contents](../Contents) \| [Previous (3.4 Modules)](04_Modules) \| [Next (3.6 Design Discussion)](06_Design_discussion) + # 3.5 Main Module This section introduces the concept of a main program or main module. @@ -22,7 +24,7 @@ class myprog { } ``` -This is the first function that is being executing when an application is launched. +This is the first function that executes when an application is launched. ### Python Main Module @@ -34,11 +36,11 @@ bash % python3 prog.py ... ``` -Whatever module you give to the interpreter at startup becomes *main*. It doesn't matter the name. +Whatever file you give to the interpreter at startup becomes *main*. It doesn't matter the name. ### `__main__` check -It is standard practice for modules that can run as a main script to use this convention: +It is standard practice for modules that run as a main script to use this convention: ```python # prog.py @@ -53,22 +55,22 @@ Statements inclosed inside the `if` statement become the *main* program. ### Main programs vs. library imports -Any file can either run as main or as a library import: +Any Python file can either run as main or as a library import: ```bash bash % python3 prog.py # Running as main ``` ```python -import prog +import prog # Running as library import ``` In both cases, `__name__` is the name of the module. However, it will only be set to `__main__` if running as main. -As a general rule, you don't want statements that are part of the main -program to execute on a library import. So, it's common to have an `if-`check in code -that might be used either way. +Usually, you don't want statements that are part of the main program +to execute on a library import. So, it's common to have an `if-`check +in code that might be used either way. ```python if __name__ == '__main__': @@ -221,7 +223,8 @@ bash % prog.py ### Script Template -Here is a common code template for Python programs that run as command-line scripts: +Finally, here is a common code template for Python programs that run +as command-line scripts: ```python #!/usr/bin/env python3 @@ -251,8 +254,9 @@ if __name__ == '__main__': ### Exercise 3.15: `main()` functions -In the file `report.py` add a `main()` function that accepts a list of command line options and produces the same output as before. -You should be able to run it interatively like this: +In the file `report.py` add a `main()` function that accepts a list of +command line options and produces the same output as before. You +should be able to run it interatively like this: ```python >>> import report @@ -280,7 +284,8 @@ Total cost: 44671.15 ### Exercise 3.16: Making Scripts -Modify the `report.py` and `pcost.py` programs so that they can execute as a script on the command line: +Modify the `report.py` and `pcost.py` programs so that they can +execute as a script on the command line: ```bash bash $ python3 report.py Data/portfolio.csv Data/prices.csv diff --git a/Notes/03_Program_organization/06_Design_discussion.md b/Notes/03_Program_organization/06_Design_discussion.md index 40a7919..5baa0ad 100644 --- a/Notes/03_Program_organization/06_Design_discussion.md +++ b/Notes/03_Program_organization/06_Design_discussion.md @@ -1,6 +1,8 @@ +[Contents](../Contents) \| [Previous (3.5 Main module)](05_Main_module) \| [Next (4 Classes)](../04_Classes_objects/00_Overview) + # 3.6 Design Discussion -In this section we consider some design decisions made in code so far. +In this section we reconsider a design decision made earlier. ### Filenames versus Iterables @@ -35,15 +37,17 @@ with open('file.csv') as f: * Which of these functions do you prefer? Why? * Which of these functions is more flexible? - ### Deep Idea: "Duck Typing" -[Duck Typing](https://en.wikipedia.org/wiki/Duck_typing) is a computer programming concept to determine whether an object can be used for a particular purpose. It is an application of the [duck test](https://en.wikipedia.org/wiki/Duck_test). +[Duck Typing](https://en.wikipedia.org/wiki/Duck_typing) is a computer +programming concept to determine whether an object can be used for a +particular purpose. It is an application of the [duck +test](https://en.wikipedia.org/wiki/Duck_test). > If it looks like a duck, swims like a duck, and quacks like a duck, then it probably is a duck. -In our previous example that reads the lines, our `read_data` expects -any iterable object. Not just the lines of a file. +In the second version of `read_data()` above, the function expects any +iterable object. Not just the lines of a file. ```python def read_data(lines): @@ -76,7 +80,7 @@ data = read_data(lines) There is considerable flexibility with this design. -*Question: Shall we embrace or fight this flexibility?* +*Question: Should we embrace or fight this flexibility?* ### Library Design Best Practices @@ -87,10 +91,10 @@ Don't restrict your options. With great flexibility comes great power. ### Exercise 3.17: From filenames to file-like objects -In this section, you worked on a file `fileparse.py` that contained a +You've now created a file `fileparse.py` that contained a function `parse_csv()`. The function worked like this: -```pycon +```python >>> import fileparse >>> portfolio = fileparse.parse_csv('Data/portfolio.csv', types=[str,int,float]) >>> @@ -103,8 +107,8 @@ with any file-like/iterable object. For example: ``` >>> import fileparse >>> import gzip ->>> with gzip.open('Data/portfolio.csv.gz', 'rt') as f: -... port = fileparse.parse_csv(f, types=[str,int,float]) +>>> with gzip.open('Data/portfolio.csv.gz', 'rt') as file: +... port = fileparse.parse_csv(file, types=[str,int,float]) ... >>> lines = ['name,shares,price', 'AA,34.23,100', 'IBM,50,91.1', 'HPE,75,45.1'] >>> port = fileparse.parse_csv(lines, types=[str,int,float]) @@ -120,8 +124,7 @@ In this new code, what happens if you pass a filename as before? >>> ``` -With flexibility comes power and with power comes responsibility. Sometimes you'll -need to be careful. +Yes, you'll need to be careful. Could you add a safety check to avoid this? ### Exercise 3.18: Fixing existing functions