zev/practical-python
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Many edits

Browse Source
This commit is contained in:
David Beazley
2020-05-28 17:41:59 -05:00
parent 9572f707b2
commit a83a9cf064
23 changed files with 467 additions and 210 deletions
Download Patch File Download Diff File Expand all files Collapse all files

87
Notes/09_Packages/01_Packages.md
View File

@@ -1,6 +1,10 @@
[Contents](../Contents) \| [Previous (8.3 Debugging)](../08_Testing_debugging/03_Debugging) \| [Next (9.2 Third Party Packages)](02_Third_party)
# 9.1 Packages
This section introduces the concept of a package.
If writing a larger program, you don't really want to organize it as a
large of collection of standalone files at the top level. This
section introduces the concept of a package.
### Modules
@@ -27,7 +31,8 @@ b = foo.spam('Hello')
### Packages vs Modules
For larger collections of code, it is common to organize modules into a package.
For larger collections of code, it is common to organize modules into
a package.
```code
# From this
@@ -43,17 +48,18 @@ porty/
fileparse.py
```
You pick a name and make a top-level directory. `porty` in the example above.
You pick a name and make a top-level directory. `porty` in the example
above (clearly picking this name is the most important first step).
Add an `__init__.py` file. It may be empty.
Add an `__init__.py` file to the directory. It may be empty.
Put your source files into it.
Put your source files into the directory.
### Using a Package
A package serves as a namespace for imports.
This means that there are multilevel imports.
This means that there are now multilevel imports.
```python
import porty.report
@@ -64,25 +70,25 @@ There are other variations of import statements.
```python
from porty import report
port = report.read_portfolio('port.csv')
port = report.read_portfolio('portfolio.csv')
from porty.report import read_portfolio
port = read_portfolio('port.csv')
port = read_portfolio('portfolio.csv')
```
### Two problems
There are two main problems with this approach.
* imports between files in the same package.
* Main scripts placed inside the package.
* imports between files in the same package break.
* Main scripts placed inside the package break.
Both break.
So, basically everything breaks. But, other than that, it works.
### Problem: Imports
Imports between files in the same package *must include the package name in the import*.
Remember the structure.
Imports between files in the same package *must now include the
package name in the import*. Remember the structure.
```code
porty/
@@ -92,7 +98,7 @@ porty/
fileparse.py
```
Import example.
Modified import example.
```python
# report.py
@@ -113,7 +119,8 @@ import fileparse # BREAKS. fileparse not found
### Relative Imports
However, you can use `.` to refer to the current package. Instead of the package name.
Instead of directly using the package name,
you can use `.` to refer to the current package.
```python
# report.py
@@ -133,16 +140,24 @@ This makes it easy to rename the package.
### Problem: Main Scripts
Running a submodule as a main script breaks.
Running a package submodule as a main script breaks.
```bash
bash $ python porty/pcost.py # BREAKS
...
```
*Reason: You are running Python on a single file and Python doesn't see the rest of the package structure correctly (`sys.path` is wrong).*
*Reason: You are running Python on a single file and Python doesn't
see the rest of the package structure correctly (`sys.path` is
wrong).*
All imports break.
All imports break. To fix this, you need to run your program in
a different way, using the `-m` option.
```bash
bash $ python -m porty.pcost # WORKS
...
```
### `__init__.py` files
@@ -156,7 +171,7 @@ from .pcost import portfolio_cost
from .report import portfolio_report
```
Makes names appear at the *top-level* when importing.
This makes names appear at the *top-level* when importing.
```python
from porty import portfolio_cost
@@ -170,16 +185,15 @@ from porty import pcost
pcost.portfolio_cost('portfolio.csv')
```
### Solution for scripts
### Another solution for scripts
Use `-m package.module` option.
As noted, you now need to use `-m package.module` to
run scripts within your package.
```bash
bash % python3 -m porty.pcost portfolio.csv
```
It will run the code in a proper package environment.
There is another alternative: Write a new top-level script.
```python
@@ -190,13 +204,24 @@ import sys
porty.pcost.main(sys.argv)
```
This script lives *outside* the package.
This script lives *outside* the package. For example, looking at the directory
structure:
```
pcost.py # top-level-script
porty/ # package directory
__init__.py
pcost.py
...
```
### Application Structure
Code organization and file structure is key to the maintainability of an application.
Code organization and file structure is key to the maintainability of
an application.
One recommended structure is the following.
There is no "one-size fits all" approach for Python. However, one
structure that works for a lot of problems is something like this.
```code
porty-app/
@@ -210,11 +235,15 @@ porty-app/
fileparse.py
```
Top-level scripts need to exist outside the code package. One level up.
The top-level `porty-app` is a container for everything else--documentation,
top-level scripts, examples, etc.
Again, top-level scripts (if any) need to exist outside the code
package. One level up.
```python
#!/usr/bin/env python3
# script.py
# porty-add/script.py
import sys
import porty
@@ -306,7 +335,7 @@ scripts, and other things. These files need to exist OUTSIDE of the
`porty/` directory you made above.
Create a new directory called `porty-app`. Move the `porty` directory
you created in part (a) into that directory. Copy the
you created in Exercise 9.1 into that directory. Copy the
`Data/portfolio.csv` and `Data/prices.csv` test files into this
directory. Additionally create a `README.txt` file with some
information about yourself. Your code should now be organized as

139
Notes/09_Packages/02_Third_party.md
View File

@@ -1,45 +1,144 @@
# 9.2 Third Party Modules
[Contents](../Contents) \| [Previous (9.1 Packages)](01_Packages) \| [Next (9.3 Distribution)](03_Distribution)
### Introduction
# 9.2 Third Party Modules
Python has a large library of built-in modules (*batteries included*).
There are even more third party modules. Check them in the [Python Package Index](https://pypi.org/) or PyPi. Or just do a Google search for a topic.
There are even more third party modules. Check them in the [Python Package Index](https://pypi.org/) or PyPi.
Or just do a Google search for a specific topic.
### Some Notable Modules
How to handle third-party dependencies is an ever-evolving topic with
Python. This section merely covers the basics to help you wrap
your brain around how it works.
* `requests`: Accessing web services.
* `numpy`, `scipy`: Arrays and vector mathematics.
* `pandas`: Stats and data analysis.
* `django`, `flask`: Web programming.
* `sqlalchemy`: Databases and ORM.
* `ipython`: Alternative interactive shell.
### The Module Search Path
`sys.path` is a directory that contains the list of all directories
checked by the `import` statement. Look at it:
```python
>>> import sys
>>> sys.path
... look at the result ...
>>>
```
If you import something and it's not located in one of those
directories, you will get an `ImportError` exception.
### Standard Library Modules
Modules from Python's standard library usually come from a location
such as `/usr/local/lib/python3.6'. You can find out for certain
by trying a short test:
```python
>>> import re
>>> re
<module 're' from '/usr/local/lib/python3.6/re.py'>
>>>
```
Simply looking at a module in the REPL is a good debugging tip
to know about. It will show you the location of the file.
### Third-party Modules
Third party modules are usually located in a dedicated
`site-packages` directory. You'll see it if you perform
the same steps as above:
```python
>>> import numpy
<module 'numpy' from '/usr/local/lib/python3.6/site-packages/numpy/__init__.py'>
>>>
```
Again, looking at a module is a good debugging tip if you're
trying to figure out why something related to `import` isn't working
as expected.
### Installing Modules
Most common classic technique: `pip`.
The most common technique for installing a third-party module is to use
`pip`. For example:
```bash
bash % python3 -m pip install packagename
```
This command will download the package and install it globally in your Python folder. Somewhere like:
```code
/usr/local/lib/python3.6/site-packages
```
This command will download the package and install it in the `site-packages`
directory.
### Problems
* You may be using an installation of Python that you don't directly control.
* A corporate approved installation
* The Python version that comes with the OS.
* You're using the Python version that comes with the OS.
* You might not have permission to install global packages in the computer.
* Your program might have unusual dependencies.
* There might be other dependencies.
### Talk about environments...
### Virtual Environments
A common solution to package installation issues is to create a
so-called "virtual environment" for yourself. Naturally, there is no
"one way" to do this--in fact, there are several competing tools and
techniques. However, if you are using a standard Python installation,
you can try typing this:
```bash
bash % python -m venv mypython
bash %
```
After a few moments of waiting, you will have a new directory
`mypython` that's your own little Python install. Within that
directory you'll find a `bin/` directory (Unix) or a `Scripts/`
directory (Windows). If you run the `activate` script found there, it
will "activate" this version of Python, making it the default `python`
command for the shell. For example:
```bash
bash % source mypython/bin/activate
(mypython) bash %
```
From here, you can now start installing Python packages for yourself.
For example:
```
(mypython) bash % python -m pip install pandas
...
```
For the purposes of experimenting and trying out different
packages, a virtual environment will usually work fine. If,
on the other hand, you're creating an application and it
has specific package dependencies, that is a slightly
different problem.
### Handling Third-Party Dependencies in Your Application
If you have written an application and it has specific third-party
dependencies, one challange concerns the creation and preservation of
the environment that includes your code and the dependencies. Sadly,
this has been an area of great confusion and frequent change over
Python's lifetime. It continues to evolve even now.
Rather than provide information that's bound to be out of date soon,
I refer you to the [Python Packaging User Guide](https://packaging.python.org).
## Exercises
(rewrite)
### Exercise 9.4 : Creating a Virtual Environment
See if you can recreate the steps of making a virtual environment and installing
pandas into it as shown above.
[Contents](../Contents) \| [Previous (9.1 Packages)](01_Packages) \| [Next (9.3 Distribution)](03_Distribution)

87
Notes/09_Packages/03_Distribution.md Normal file
View File

@@ -0,0 +1,87 @@
[Contents](../Contents) \| [Previous (9.2 Third Party Packages)](02_Third_party) \| [Next (The End)](TheEnd)
# 9.3 Distribution
At some point you might want to give your code to someone else, possibly just a co-worker.
This section gives the most basic technique of doing that. For more detailed
information, you'll need to consult the [Python Packaging User Guide](https://packaging.python.org).
### Creating a setup.py file
Add a `setup.py` file to the top-level of your project directory.
```python
# setup.py
import setuptools
setuptools.setup(
name="porty",
version="0.0.1",
author="Your Name",
author_email="you@example.com",
description="Practical Python Code",
packages=setuptools.find_packages(),
)
```
### Creating MANIFEST.in
If there are additional files associated with your project, specify them with a `MANIFEST.in` file.
For example:
```
# MANIFEST.in
include *.csv
```
Put the `MANIFEST.in` file in the same directory as `setup.py`.
### Creating a source distribution
To create a distribution of your code, use the `setup.py` file. For example:
```
bash % python setup.py sdist
```
This will create a `.tar.gz` or `.zip` file in the directory `dist/`. That file is something
that you can now give away to others.
### Installing your code
Others can install your Python code using `pip` in the same way that they do for other
packages. They simply need to supply the file created in the previous step.
For example:
```
bash % python -m pip install porty-0.0.1.tar.gz
```
### Commentary
The steps above describe the absolute most minimal basics of creating
a package of Python code that you can give to another person. In
reality, it can be much more complicated depending on third-party
dependencies, whether or not your application includes foreign code
(i.e., C/C++), and so forth. Covering that is outside the scope of
this course. We've only taken a tiny first step.
## Exercises
### Exercise 9.5: Make a package
Take the `porty-app/` code you created for Exercise 9.3 and see if you
can recreate the steps described here. Specifically, add a `setup.py`
file and a `MANIFEST.in` file to the top-level directory.
Create a source distribution file by running `python setup.py sdist`.
As a final step, see if you can install your package into a Python
virtual environment.
[Contents](../Contents) \| [Previous (9.2 Third Party Packages)](02_Third_party) \| [Next (The End)](TheEnd)

12
Notes/09_Packages/TheEnd.md Normal file
View File

@@ -0,0 +1,12 @@
# The End!
You've made it to the end of the course. Thanks for your time and your attention.
May your future Python hacking be fun and productive!
I'm always happy to get feedback. You can find me at [https://dabeaz.com](https://dabeaz.com)
or on Twitter at [@dabeaz](https://twitter.com/dabeaz).
- David Beazley
[Contents](../Contents) \| [Home](../..)