Python Basics

for programmers…

Disclaimer: I am not a python professional but an enthusiast.

History

Implemented in 1989 by Guido van Rossum based on ABC language

● Named after Monty Python's Flying Circus
● Interpretive (byte-code & virtual machine)
● Flavors
○ Incompatibility with unicode/string and some constructs
○ < 2.7 (do not use or look for stable releases)
○ 2.7+ - 2.7.11 as of now
○ 3.0+ : 3.6.1 as of now, (3.7 release schedule)
● Open source, with huge library and lots of contributions
■ https://github.com/python/cpython
○ Networking, web, database, desktop, GUI, scientific, ...
● Superior in terms of density (less typing) and clarity (very readable code)
○ Easy-to-learn/read/maintain
● Continually enhanced through PEPs
(Python Enhancement Proposals)
○ https://www.python.org/dev/peps/
● Python can be Embedded and Extended (C/C++)
Install Python

https://www.python.org

https://conda.io/miniconda.html

Faster Python
● PyPy
● Cython (optimising static compiler)
Online IDEs REPL (Read–Eval–Print Loop)

https://try.jupyter.org/

http://www.skulpt.org/

https://repl.it/ https://www.python.org/shell/ https://www.pythonanywhere.com/

Python Interactive Shells

Python shells IPython/Jupyter

$ python # Terminal
$ python2 $ ipython
$ python3
$ python3.6 # Web
$ ipython notebook
IDLE
qtconsole
$ idle3
$ python3 -m idlelib
$ ipython qtconsole
bpython

$ pbython

ptpython

$ ptython
Python IDEs

Editors
● SublimeText 3

Python IDE
● IDLE ($ python3 -m idlelib # Or idle3)
● spyder
● PyDev (eclipse)
● pycharm
● WingIDE
Command Line Arguments and Environment Variables

Useful command line arguments:

$ python -h # python2 -h # python3 -h

usage: python [option] ... [-c cmd | -m mod | file | -] [arg] ...

-B : don't write .py[co] files on import; also PYTHONDONTWRITEBYTECODE=x py3:(__pycache__/*.cpython-36.pyc)

-c cmd : program passed in as string (terminates option list)
-E : ignore PYTHON* environment variables (such as PYTHONPATH)
-i : inspect interactively after running script; forces a prompt even
if stdin does not appear to be a terminal; also PYTHONINSPECT=x
-m mod : run library module as a script (terminates option list)
-O : optimize generated bytecode slightly; also PYTHONOPTIMIZE=x *.pyo (__pycache__/*.cpython-36.opt-1.pyc)
-OO : remove doc-strings in addition to the -O optimizations *.pyo (__pycache__/*.cpython-36.opt-2.pyc)
-s : don't add user site directory to sys.path; also PYTHONNOUSERSITE (import site)
-3 : warn about Python 3.x incompatibilities that 2to3 cannot trivially fix

Useful environment variables:

PYTHONPATH : ':'-separated list of directories prefixed to the

default module search path. The result is sys.path.
Python module extensions: (.py, .pyw) → (main scripts do not need the extension)
Python REPL (Read–Eval–Print Loop)

Python Shell Prompts # Comments start with hash/pound sign

>>> # sys.ps1
# Ways to exit interpreter
... # sys.ps2 exit() or quit() or Ctrl-D (i.e. EOF) to exit
raise SystemExit
Get names defined in a module import sys; sys.exit()
>>> dir()
['__builtins__', '__doc__', '__loader__', '__name__', '__package__', '__spec__'] # py3
>>> dir(__builtins__)
['ArithmeticError', 'AssertionError', 'AttributeError', 'BaseException', 'BlockingIOError', 'BrokenPipeError', 'BufferError', 'BytesWarning', 'ChildProcessError',
'ConnectionAbortedError', 'ConnectionError', 'ConnectionRefusedError', 'ConnectionResetError', 'DeprecationWarning', 'EOFError', 'Ellipsis', 'EnvironmentError',
'Exception', 'False', 'FileExistsError', 'FileNotFoundError', 'FloatingPointError', 'FutureWarning', 'GeneratorExit', 'IOError', 'ImportError', 'ImportWarning',
'IndentationError', 'IndexError', 'InterruptedError', 'IsADirectoryError', 'KeyError', 'KeyboardInterrupt', 'LookupError', 'MemoryError', 'ModuleNotFoundError',
'NameError', 'None', 'NotADirectoryError', 'NotImplemented', 'NotImplementedError', 'OSError', 'OverflowError', 'PendingDeprecationWarning', 'PermissionError',
'ProcessLookupError', 'RecursionError', 'ReferenceError', 'ResourceWarning', 'RuntimeError', 'RuntimeWarning', 'StopAsyncIteration', 'StopIteration', 'SyntaxError',
'SyntaxWarning', 'SystemError', 'SystemExit', 'TabError', 'TimeoutError', 'True', 'TypeError', 'UnboundLocalError', 'UnicodeDecodeError', 'UnicodeEncodeError',
'UnicodeError', 'UnicodeTranslateError', 'UnicodeWarning', 'UserWarning', 'ValueError', 'Warning', 'ZeroDivisionError', '__build_class__', '__debug__', '__doc__',
'__import__', '__loader__', '__name__', '__package__', '__spec__', 'abs', 'all', 'any', 'ascii', 'bin', 'bool', 'bytearray', 'bytes', 'callable', 'chr', 'classmethod',
'compile', 'complex', 'copyright', 'credits', 'delattr', 'dict', 'dir', 'divmod', 'enumerate', 'eval', 'exec', 'exit', 'filter', 'float', 'format', 'frozenset',
'getattr', 'globals', 'hasattr', 'hash', 'help', 'hex', 'id', 'input', 'int', 'isinstance', 'issubclass', 'iter', 'len', 'license', 'list', 'locals', 'map', 'max',
'memoryview', 'min', 'next', 'object', 'oct', 'open', 'ord', 'pow', 'print', 'property', 'quit', 'range', 'repr', 'reversed', 'round', 'set', 'setattr', 'slice',
'sorted', 'staticmethod', 'str', 'sum', 'super', 'tuple', 'type', 'vars', 'zip']
Builtin functions py2: print → statement, py3: print → function
Easter eggs:
Get help (pydoc)
>> help() >> import __hello__
Hello world!
>> help(‘name’) # help(name) → name should be in scope >> import antigravity
>> from __future__ import braces
PEP 20 -- The Zen of Python File "<stdin>", line 1
SyntaxError: not a chance
>> import this
Python Interactive Shell (continued...)

Debugging tools

● pdb

$ python3 -m pdb myscript.py

>>> import pdb

>>> import mymodule
>>> pdb.run('mymodule.test()')

objbrowser

>>> from objbrowser import browse

>>> browse(locals())

Object size

>>> import sys

>>> sys.getsizeof(int)
Command Line Arguments

Help https://docs.python.org/2/ Python debugger

$ pydoc https://docs.python.org/3/ $ python -m pdb script.py
$ python -m pydoc <name> Convert py2 to py3
$ python -m pydoc -p 8080 $ 2to3
$ python -m lib2to3
Simple HTTP Server
$ python2 -m SimpleHTTPServer Base64 encode/decode
$ python3 -m http.server $ python -m base64 -h
$ echo "This is test" | python -m base64 -
VGhpcyBpcyB0ZXN0Cg==
IDLE (Python Integrated DeveLopment Environment)
$ echo "VGhpcyBpcyB0ZXN0Cg==" | python -m base64 -d -
$ idle This is test
$ python -m idlelib.idle

Display sys.path (PYTHONPATH)

Alternate shell (IPython / Jupyter) $ /opt/tools/wh/dtd/RHE64-6/python/3.6/bin/python3 -m site
$ ipython sys.path = [
'/home/akhailta/work/Amal/Python/test',
$ jupyter qtconsole '$PYTHONHOME/lib/python36.zip',
$ bpython '$PYTHONHOME/lib/python3.6',
'$PYTHONHOME/lib/python3.6/lib-dynload',
'/home/akhailta/.local/lib/python3.6/site-packages',
Pretty print JSON '$PYTHONHOME/lib/python3.6/site-packages',
'$PYTHONHOME/lib/python3.6/site-packages/Sphinx-1.5.6-py3.6.egg',
$ cat file.json | python -m json.tool '$PYTHONHOME/lib/python3.6/site-packages/setuptools-27.2.0-py3.6.egg',
'$PYTHONHOME/lib/python3.6/site-packages/rst2pdf-0.93.dev_r0-py3.6.egg',
'$PYTHONHOME/lib/python3.6/site-packages/mccabe-0.6.1-py3.6.egg',
'$PYTHONHOME/lib/python3.6/site-packages/pytest_cache-1.0-py3.6.egg',
Send file as attachment (uu encode/decode) '$PYTHONHOME/lib/python3.6/site-packages/pytest_pep8-1.0.6-py3.6.egg',
]
$ python -m uu -h # help on uu command USER_BASE: '/home/akhailta/.local' (exists)
$ cat file | python -m uu | python -m uu -d USER_SITE: '/home/akhailta/.local/lib/python3.6/site-packages' (exists)

$ cat file | python -m uu | /usr/sbin/sendmail [email protected]

Python Package Manager (PyPM)

Commonly used

● easy_install (setuptools, distutils - default on old systems; do NOT use)

○ No uninstall
○ e.g: easy_install hexdump
● pip (comes pre-installed with Python 3)
○ Easier to use and has uninstall
○ e.g
■ pip search
■ pip install --user --upgrade hexdump ; # ~/.local/ or %APPDATA%\Python
■ pip uninstall hexdump
■ pip list
● Continuum Analytics conda (miniconda)
○ Provides binary packages as well
○ e.g.:
■ conda search pyqt
■ conda install pyqt
■ conda uninstall pyqt
■ conda update --all
■ conda list
Python Package Index (PyPI)

PYPI: https://pypi.python.org/pypi
Warehouse: https://pypi.io/ (currently in pre-production)

The Python Package Index is a repository of software for the Python programming language

● easy_install and pip use PYPI repository

● User contribution

Conda repositories

● Official repository (used by conda) - http://repo.continuum.io/pkgs/

● CAUTION - Unofficial (user contributed) - https://anaconda.org/
Virtualenv

Manage python package

● Versions
○ Avoid conflicts
● Dependencies
○ Avoid conflicts
● Permissions
○ Allow user install
● Python2:
○ $ pip install virtualenv
○ $ virtualenv venv2
○ $ . venv2/bin/activate # or source venv2/bin/activate.csh
○ $ pip install …
○ $ deactivate

● Available in py3 by default (conda does not have ensurepip module and pip installation)
○ $ pyvenv --without-pip venv3
○ $ python3 -m venv --without-pip venv3
○ $ . venv3/bin/activate # or source venv3/bin/activate.csh
○ $ wget -O - https://bootstrap.pypa.io/get-pip.py | python
○ $ pip instal ...
○ $ deactivate

● Allows separation of modules from standard site-package

Python Docstrings (PEP 257)

PEP 257 -- Python Docstring Conventions

A docstring is a string literal that occurs as the first statement in a module, function, class, or method
definition.
● Such a docstring becomes the __doc__ special attribute of that object.
● help and pydoc use this attribute
● All modules should normally have docstrings
● All functions and classes exported by a module should also have docstrings
● Public methods (including the __init__ constructor) should also have docstrings
● A package may be documented in the module docstring of the __init__.py file in the package directory.
● For consistency, always use """triple double quotes""" around docstrings.
● Use r"""raw triple double quotes""" if you use any backslashes in your docstrings.
● For Unicode docstrings, use u"""Unicode triple-quoted strings""" (from __future__ import unicode_literals)

pep8 / pycodestyle: checks code style (--ignore=...)

$ conda/pip install pep8

$ pep8 [--statistics] [--show-source] [file name or directory name]
$ python -m pep8 [--statistics] [--show-source] [file name or directory name]
Python Docstrings (continued...)

#!/usr/bin/env python $ python

# -*- coding: utf-8 -*- >>> __name__
'__main__'
# prog.py >>> dir()
['__builtins__', '__doc__', '__name__', '__package__']
"""Module docstring""" >>> import prog
>>> dir()
import sys ['__builtins__', '__doc__', '__name__', '__package__', 'prog']
>>> dir(prog)
['__builtins__', '__doc__', '__file__', '__name__', '__package__',
def main(args): 'main', 'sys']
"""Main entry point""" >>> prog.__doc__
print(sys.version) 'module docstring'
print() >>> prog.main.__doc__
print(' __name__ :', __name__) 'Main entry point'
print(' __doc__ :', __doc__) >>> help(prog)
print('main.__name__ :', main.__name__) Help on module prog:
print('main.__doc__ :', main.__doc__)
NAME
__main__ - Module docstring
if __name__ == '__main__':
main() CLASSES
builtins.object
A

class A(builtins.object)
| Class A.

FUNCTIONS
main(args)
Main entry point
FILE
/home/akhailta/work/Amal/Python/prog.py
Underscore: _, Double-Underscore (dunder, double-under): __

● Single Leading Underscore: _var

○ Internal use (does not mean private as in Java)
○ Convention only/programmer hint
○ Will not be imported (import *) unless it is part of __all__
● Single Trailing Underscore: var_
○ Avoid name clashes with python keywords or standard modules
● Double Leading Underscore: __var
○ Mangled by python (_CLASSNAME__var)
○ Avoid name conflicts in subclasses
● Double Leading and Trailing Underscore: __var__
○ Magic methods
○ Normally reserved for special use (__init__, __doc__, …)
● Single Underscore: _
for _ in range(10):
○ Use for variables without a name or unimportant print('Hello')
○ Used in REPL to store result of last expression evaluation
x, _, z = (10, 20, 'abc')

https://dbader.org/blog/meaning-of-underscores-in-python
Python Modules

Module (modules search: current directory, sys.path - PYTHONPATH)

● Allows organizing Python code logically
○ Group related code into a module makes the code easier to understand and use
● Is a Python object with arbitrarily named attributes that you can bind and reference.
● Is a file consisting of Python code
○ Can define functions, classes and variables
○ Can also include runnable code.

$ cat mod.py
def hello(name):
print(‘Hello’, name)

>>> import mod

>>> mod.hello(‘Python’)

>>> from mod import hello

>>> hello(‘Python’)

>>> from mod import *

>>> hello(‘Python’)
Python Packages

Package
● Is a way of structuring Python’s module namespace by using “dotted module names”
● Is a hierarchical file directory structure that defines a single Python application environment that
consists of modules and subpackages and sub-subpackages, and so on.
sound/ # Top-level package
__init__.py # Initialize the sound package
formats/ # Subpackage for file format conversions
__init__.py
wavread.py
effects/ # Subpackage for sound effects
__init__.py # __all__ = ["echo", "surround", "reverse"]
echo.py
filters/ # Subpackage for filters
__init__.py
...

>>> import sound.effects.echo

>>> sound.effects.echo.echofilter(input, output, delay=0.7, atten=4)

>>> from sound.effects import echo

>>> echo.echofilter(input, output, delay=0.7, atten=4)

>>> from sound.effects.echo import echofilter

>>> echofilter(input, output, delay=0.7, atten=4)
Python Packages (continued)

Importing * From a Package

● The import statement uses the following convention: if a package’s __init__.py code defines a list named __all__, it is taken to
be the list of module names that should be imported when from package import * is encountered.

$ cat sound/effects/__init__.py

__all__ = ["echo", "surround", "reverse"]

# import the three named submodules of the sound package.

from sound.effects import *

Intra-package references (relative imports)

from . import echo
from .. import formats
from ..filters import equalizer
Namespaces and Scopes

● Variables are names (identifiers) that map to objects.

● A namespace is a dictionary of variable names (keys) and their corresponding objects (values).
● A Python statement can access variables in a local namespace and in the global namespace.
○ If a local and a global variable have the same name, the local variable shadows the global variable.
○ global VarName → VarName is a global variable
● dir()
● globals() → dictionary of global variables
>>> globals() *py2
{'__builtins__': <module '__builtin__' (built-in)>, '__name__': '__main__', '__doc__': None, '__package__': None}

>>> globals() *py3 (keys are sorted!)

{'__builtins__': <module 'builtins' (built-in)>, '__doc__': None, '__package__': None, '__name__': '__main__',
'__spec__': None, '__loader__': <class '_frozen_importlib.BuiltinImporter'>}

>>> x = 0
>>> globals()['x'] = 10
>>> x
10
locals() → (copy of) dictionary of local variables (read-only)
Namespaces and Scopes (continued)

(B) builtins scope/namespace

# builtin scope (B)
reference object
open, range, len, SyntaxError
open function
range function
# global (G)
…
x = 1
SystemExit class
s = 'test'
(G) global scope
class Foo:
# class scope (C) reference object
y = 4 x int(1)
s str('test')
def f1()
# enclosing function (E) (C) class scope
x = 'abc' # local reference object
global s # s top-level y int(4)

def f2(): (C) enclosing function scope

# local inner function (L)
nonlocal x # x in f1() reference object
global s # s top-level x str('abc')
y = 2
(L) local inner function scope
reference object
y int(2)
Namespaces and Scopes (continued)

# scopes.py $ python3 scopes.py

# global (G) globals(): {'s': 'test', 'x': 1, …}
x = 1 globals(): {'s': 'test', 'x': 2, …}
s = 'test' f1() locals: {'x': 'abc'}
print('globals():', globals()) f1() locals: {'x': 'abc'}
globals()['x'] = 2 f2() locals: {'x': 'abc', 'y': 2}
print('globals():', globals()) f2() locals: {'x': 'abc', 'y': 2}

def f1()
# enclosing function (E)
x = 'abc' # local
global s # s top-level
print('f1() locals:', locals())
locals()['x'] = 93
print('f1() locals:', locals())

def f2():
# local inner function (L)
nonlocal x # x in f1()
global s # s top-level
y = 2
print('f2() locals:', locals())
locals()['y'] = 93
print('f2() locals:', locals())

f2()
f1()
Python Data Model (Objects)
Object

● Everything is an object
○ Object are created when needed and garbage collected when not referenced (reference counting)
● Objects have
○ Identity - is operator compares the id() of two objects (unique integer/long integer - address)
○ Type - type() returns object type # Py3 # Py2
○ Value - mutable/immutable # #
>>> a = 6 >>> a = 6
>>> a >>> a
>>> type(4) 6 6
<class 'int'> >>> >>>
>>> type(4.5) >>> print(type(a)) >>> print(type(a))
<class 'float'> <class 'int'> <type 'int'>
>>> type('abc') 5 0x7fef64776720 >>> >>>
<class 'str'> >>> a = int(6) >>> a = int(6)
>>> type([1,2,3]) >>> a >>> a
<class 'list'> 6 6
>>> type((1,2,3)) a 6 0x7fef64776740 >>> >>>
<class 'tuple'> >>> print(a.__class__) >>> print(a.__class__)
>>> type({1,2,3}) <class 'int'> <type 'int'>
<class 'set'> >>> hex(id(a)) >>> hex(id(a))
>>> type({'a':1,'b':2}) '0x7fef64776740' '0x7f97b9acd740'
<class 'dict'> >>> hex(id(6)) >>> hex(id(6))
>>> type(True) '0x7fef64776740' '0x7f97b9acd740'
<class 'bool'> >>> hex(id(5)) >>> hex(id(5))
'0x7fef64776720' '0x7f97b9acd720'
Variables

Assignment

a_number = 100
a_string = 'Error finding the results!'

Multiple assignment

a = b = c = 1
x, y, z = 55, 'string', -21

Delete variable reference

del var
Types

● None
● NotImplemented
● Ellipsis (...)
● Numbers (class numbers.Number)
○ Integral (class numbers.Integral)
■ Integer (int) :2
■ Boolean (bool) : True/False
○ Real (class numbers.Real) (float) : 3.14
○ Complex (complex) : 3.14e-10j
● Sequences
○ Immutable sequences
■ Strings/Unicode : ‘abc’, u"\N{BLACK SPADE SUIT}", ‘a’, “ABC”, ‘’’xyz’’’, “””XuX”””
■ Tuples : (33, ‘a’, [4, 7])
■ Bytes : b'abc'
○ Mutable sequences
■ Lists : [‘1’, ‘a’, [2, 3]]
■ Byte Arrays : bytearray([0x13, 0x00, 0x00, 0x00, 0x08, 0x00])
● Set types
○ Sets : {1, 2, 3, ‘a’} empty set: set()
○ Frozenset (immutable set) : frozenset([1, 2, 3, ‘a’])
● Mappings
○ Dictionaries : {‘a’: 1, ‘b’: [5, 6]}
Types (continued…)

Scalars Non-scalars
● int (2) ● sequences
● float (3.14) ○ str ‘a’, “ABC”, ‘’’xyz’’’, “””XuX”””
● bool (True, False) ■ → immutable
● NonType (None) ○ list [‘1’, ‘a’, [2, 3]]
○ tuple (33, ‘a’, [4, 7])
Operators ■ → immutable
● Assign: = ● dict {‘a’: 1, ‘b’: [5, 6]}
● Arithmetic: + - * / // % ** += -= *= /= ● set, frozenset {1, 2, 3, ‘a’}
○ 3/2=1 # *py2 ● file
○ from __future__ import division ● contextmanager (with)
○ 3 / 2 = 1.5, 3 // 2 = 1 # or py3 ● module, function, method
● Conditional: > >= < <= == != <> ● class
● Membership/identity tests: in, not in, is, is not
● Logical (boolean): and or not sequence operators:
● Bitwise: & | ^ << >> ~ ● len min max index count
● Lambda expression: lambda ● insert append pop remove reverse sort
● Conditional expression: if – else ● in not in + * [] :
● Subscription, slicing, call, attribute reference ○ [start : end : step]
○ x[index], x[index:index], x(arguments...), x.attribute ○ [::-1]
Types (continued…)

Data type conversion functions

Function Description

class int(x [,base]) Converts x to an integer. base specifies the base if x is a string.
class float(x) Converts x to a floating-point number.
class complex(real [,imag]) Creates a complex number.
class str(x) Converts object x to a string representation.
repr(x) Converts object x to an expression string.
eval(str, globals=None, locals=None) Evaluates a string and returns an object.
class tuple(s) Converts s to a tuple.
class list(s) Converts s to a list.
class set(s) Converts s to a set.
class dict(d) Creates a dictionary. d must be a sequence of (key,value) tuples.
class frozenset(s) Converts s to a frozen set.
chr(x) Converts an integer to a character.
unichr(x) Converts an integer to a Unicode character.
ord(x) Converts a single character to its integer value.
hex(x) Converts an integer to a hexadecimal string.
oct(x) Converts an integer to an octal string.
bin(x) Converts an integer to an binary string.
*py2
*py3
Types (continued…)

● Callable
○ User-defined functions - attrs on next page
○ Instance methods - attrs : __self__, __func__, __doc__ (__func__.__doc__), __name (__func__.__name__), __module__
○ Generator functions (yield, __next__())
○ Coroutine functions *py3
○ Built-in functions - attrs: __doc__, __name__, __module__
○ Built-in methods
○ Class types (new style classes - inherit from object)
○ Classic classes *py2
○ Class instances (implement __call__())
● Modules (import)
● Classes (new style and classic)
○ __name__, __module__, __dict__ - C.x == C.__dict__["x"] -, __bases__, __doc__
● Class instances
○ __class__, __dict__, __setattr__(), __getattr__(), __delattr__()
● I/O or File - sys.stdin, sys.stdout and sys.stderr, open()
● Internal types
○ Code (compiled byte-code)
○ Frame (execution frame, stack)
○ Traceback
○ Slices (a[i:j:step], a[i:j, k:l], or a[..., i:j], slice())
○ Static method
○ Class method
Types (continued…)

User-defined function attributes

Attribute Meaning

__doc__ The function’s documentation string, or None if unavailable; not inherited by subclasses Writable

__name__ The function’s name Writable

__qualname__ The function’s qualified name New in version 3.3. Writable

__module__ The name of the module the function was defined in, or None if unavailable. Writable

__defaults__ A tuple containing default argument values for those arguments that have defaults, or None if no arguments have a default value Writable

__code__ The code object representing the compiled function body. Writable

__globals__ A reference to the dictionary that holds the function’s global variables — the global namespace of the module in which the function Read-only
was defined.

__dict__ The namespace supporting arbitrary function attributes. Writable

__closure__ None or a tuple of cells that contain bindings for the function’s free variables. Read-only

__annotations__ A dict containing annotations of parameters. The keys of the dict are the parameter names, and 'return' for the return annotation, if Writable
provided.

__kwdefaults__ A dict containing defaults for keyword-only parameters. Writable

Sequences

Common sequences (str, list, tuple) methods

● 'count'
● 'cmp' → __builtins__
● 'index'
● 'max' → __builtins__
● 'min' → __builtins__
● 'len' → __builtins__

*py2
*py3
Sequences - class str (immutable)

String operations
● 'a', "ABC", '''xyz''', """XuX"""
○ r/R → raw string
○ u/U → unicode string
○ b/B → byte string
● class str methods
○ 'capitalize', 'casefold', 'center', 'count', 'decode', 'encode', 'endswith', 'expandtabs',
'find', 'format', 'format_map', 'index', 'isalnum', 'isalpha', 'isdecimal', 'isdigit',
'isidentifier', 'islower', 'isnumeric', 'isprintable', 'isspace', 'istitle', 'isupper',
'join', 'ljust', 'lower', 'lstrip', 'maketrans', 'partition', 'replace', 'rfind', 'rindex',
'rjust', 'rpartition', 'rsplit', 'rstrip', 'split', 'splitlines', 'startswith', 'strip',
'swapcase', 'title', 'translate', 'upper', 'zfill'
● len() → length
● [] → slicing, sequence operations s[5:], s[:-5], s[::-1], s[::2]
● + → concatenate first_name + ' ' + last_name
● * → repetition '=' * 80
● in, not in → membership 'a' in 'Big str', 'a' not in 'Big str'
● for c in s: ... → iteration
● `obj` → repr(obj)
*py2
● % → format string (similar to c or java) 'x = %d, s = "%s"' % (10, 'abc)
*py3
Sequences - class list

List operations
● [1, 2, [4, 'aaaa'], x, dd]
● class list methods
○ 'append', 'clear', 'copy', 'count', 'extend', 'index', 'insert', 'pop', 'remove', 'reverse',
'sort'
● list() → convert sequence/iterable to list
● len() → length
● [] → slicing, sequence operations l[5:], l[:-5], l[:], l[::-1], l[::2]
● del l[?] → delete list element
● + → concatenation (extend)
● * → repetition
● in → membership
● for x in l: ... → iteration

TIP: Reverse a list:

>>> lst = [1, 2, 3] # → [3, 2, 1]
>>> lst.reverse() # in-place
>>> lst_rev_copy = lst[::-1] # copy *py2
>>> lst_new = list(reversed(lst)) # iterator *py3
Sequences - class tuple (immutable)

Tuple operations
● () (1, ) ('xyz', 123, 'abc', obj1) 1, 2, 3
● class tuple methods
○ 'count', 'index'
● tuple() → convert sequence to tuple
● len() → length
● [] → slicing, sequence operations t[5:], t[:-5], t[:], t[::-1], t[::2]
● + → concatenation
● * → repetition
● in → membership
● for x in l: ... → iteration

*py2
*py3
class dictionary

Dictionary (map, hash, …)

● {'FirstName': 'Jack', 'LastName': 'London', 'Age': 99, }
● Keys are unique and immutable
● class dict methods
○ 'clear', 'copy', 'fromkeys', 'get', 'has_key', 'items', 'iteritems', 'iterkeys',
'itervalues', 'keys', 'pop', 'popitem', 'setdefault', 'update', 'values', 'viewitems',
'viewkeys', 'viewvalues'
● d['key'] → access value
● d['key'] = v → update value
● len() → length
● del d[‘key’] → delete dictionary element
● in → membership
● for k, v in d.items(): ... → iteration
● for k in d.keys(): ... → iteration
● for v in d.values(): ... → iteration

*py2
*py3
class set/frozenset

Set
● {'Apple', 'Orange', 'Pear'} set(['Apple', 'Orange', 'Pear'])
● class set methods
○ 'add', 'clear', 'copy', 'difference', 'difference_update', 'discard', 'intersection',
'intersection_update', 'isdisjoint', 'issubset', 'issuperset', 'pop', 'remove',
'symmetric_difference', 'symmetric_difference_update', 'union', 'update'
● class frozenset methods
○ 'copy', 'difference', 'intersection', 'isdisjoint', 'issubset', 'issuperset',
'symmetric_difference', 'union'
● set() / frozenset() → convert sequence/iterable to set/frozenset
● s['key'] → access value
● len() → length
● del s['key'] → delete set element
issubset(other) == set <= other
● in → membership set < other == set <= other and set != other
issuperset(other) == set >= other
● for x in s: ... → iteration set > other == set >= other and set != other
● Operator equivalents union(other, ...)
intersection(other, ...)
==
==
set
set
| other | ...
& other & ...
difference(other, ...) == set - other - ...
symmetric_difference(other) == set ^ other
update(other, ...) == set |= other | ...
intersection_update(other, ...) == set &= other & ...
difference_update(other, ...) == set -= other | ...
symmetric_difference_update(other) == set ^= other
Python Language Reference
Language Reference

Notes
● DO NOT USE TABS AND DO NOT MIX TABS AND SPACES
● Use 4 spaces for indents
○ Code block with same indentation is called a suite
if expression :
suite
elif expression :
suite
else :
suite
○ Same indentation in a suite
○ Don’t mix space/tab
● Explicit line joining using ‘\’ (rarely used, not recommended)
● Comments start with ‘#’
● Multiple statements can be place on a single line joined with ‘;’ (not recommended)
● Python 2 Grammar, Python 3 Grammar
● PEP8 : Style Guide for Python Code - pycodestyle
○ PEP8 online check
○ Elements of Python Style
● PEP257: Python Docstring Conventions - pydocstyle
● Pyflakes - passive checker of Python programs
● Pylint - lint
Language Reference (continued…)

PEP8 - Style Guide for Python

● Indentation → 4 spaces per level
● Maximum line length → 79 (\ to break long lines)
● Top-level functions and classes separated by two blank lines
● Use UTF-8 encoding
● Imports should be on separate lines
● Use single-quote or double-quote for all strings and triple double-quotes for docstrings
● Avoid whitespace before , or : or immediately after/before parentheses (), brackets [] or braces {}
● Avoid trailing whitespace
● Surround binary operators with single space
● Use more block/accurate comments, use inline comments sparingly
● For version control use the following after module docstring
__version__ = "$Revision$"
# $Source$
● Module names → all lower case with optional _
● Class names → CapWord (CamelCase)
● Function names → all lowercase, words separated with _
○ Always use self for the first argument to instance methods.
○ Always use cls for the first argument to class methods.
● Module-level constants → ALL_CAPS with _

Functions can be annotated based on PEP 484 (Type Hints)

Reserved classes of identifiers
● _*
○ Not imported by from module import *.
○ The special identifier _ is used in the interactive interpreter to store the result of the last evaluation; it is stored in
the __builtin__ module. When not in interactive mode, _ has no special meaning and is not defined.
● __*__
○ System-defined names. These names are defined by the interpreter and its implementation (including the standard
library).
● __*
○ Class-private names.
○ Names in this category, when used within the context of a class definition, are re-written to use a mangled form to
help avoid name clashes between “private” attributes of base and derived classes.

>>> import math

>>> dir(math)
['__doc__', '__file__', '__name__', '__package__', 'acos', 'acosh', 'asin', 'asinh', 'atan', 'atan2', 'atanh', 'ceil', 'copysign', 'cos', 'cosh',
'degrees', 'e', 'erf', 'erfc', 'exp', 'expm1', 'fabs', 'factorial', 'floor', 'fmod', 'frexp', 'fsum', 'gamma', 'hypot', 'isinf', 'isnan', 'ldexp',
'lgamma', 'log', 'log10', 'log1p', 'modf', 'pi', 'pow', 'radians', 'sin', 'sinh', 'sqrt', 'tan', 'tanh', 'trunc']
>>> math.__name__
'math'
import math, pprint
>>> math.__file__
pprint.pprint(dir(math))
'.../lib/python2.7/lib-dynload/math.so'
>>> math.__doc__
'This module is always available. It provides access to the\nmathematical functions defined by the C standard.'
for k, v in math.__dict__.items():
print("%-11s : %s" % (k, repr(v)))
Language Reference (continued…)

Keywords (case sensitive)

and as assert break class continue def del elif else except exec finally for from global if import in is lambda nonlocal not
or pass raise return try while with yield

>>> import keyword; print(keyword.kwlist) # python -c "import keyword; print(keyword.kwlist)"

Constants
False
True
None
NotImplemented
Ellipsis → …, slicing [?:?:]
__debug__ → True if python -O else: False

Constants added by ‘site’ module (automatically imported unless python -s is used)

quit([code=None])
exit([code=None])
copyright
license
credits
Language Reference (continued…)

String Literals 'single-quotes'

"double-quotes"
'''triple single-quotes
stringliteral ::= [stringprefix](shortstring | longstring) can span multiple lines'''
stringprefix ::= "r" | "u" | "ur" | "R" | "U" | "UR" | "Ur" | "uR" """triple double-quotes
| "b" | "B" | "br" | "Br" | "bR" | "BR" can span multiple lines"""
shortstring ::= "'" shortstringitem* "'" | '"' shortstringitem* '"'
r'\n is written as is'
longstring ::= "'''" longstringitem* "'''"
| '"""' longstringitem* '"""' # b B is ignored in python2
shortstringitem ::= shortstringchar | escapeseq b'\x7fELF\x01\x01\x01\x00'
longstringitem ::= longstringchar | escapeseq
shortstringchar ::= <any source character except "\" or newline or the quote> # raw + unicode
ur"\u0062\n"
longstringchar ::= <any source character except "\">
escapeseq ::= "\" <any ASCII character> # Unicode (py3: unicode)
U'text'
String Concatenation: + or multiple string literals side-by-side # /
u"\N{SOLIDUS}
Escape sequences similar to C++/Java # ♠
u"\N{BLACK SPADE SUIT}"

\newline, \\, \', \", \a, \b, \c, \f, \n, \r, \s, \t, \v, \ooo, \xhh Note: Adjacent strings are
\N{name}, \uxxxx, \Uxxxxxxxx concatenated:
\cx or \C-x → Control-x 'ABC' "DEF" → "ABCDEF"
\M-\C-x → Meta-Control-x
Language Reference (continued…)

>>> my_string = "Hello World" Py2 (str vs unicode)

>>> type(my_string) ● implicitly decodes (easy, but painful
<type 'str'> when fails)
>>> my_unicode = u"Hi \u2119\u01b4\u2602\u210c\xf8\u1f24"
>>> type(my_unicode) >>> u"Hello " + "world"
<type 'unicode'> u'Hello world'
>>> len(my_unicode)
9 >>> u"Hello " + ("world".decode("ascii"))
u'Hello world'
>>> my_utf8 = my_unicode.encode('utf-8')
>>> len(my_utf8) Py3 (str vs bytes)
19
● won’t implicitly change bytes ↔
>>> my_utf8 unicode
'Hi \xe2\x84\x99\xc6\xb4\xe2\x98\x82\xe2\x84\x8c\xc3\xb8\xe1\xbc\xa4' ● Mixing bytes/unicode is painful

>>> my_utf8.decode('utf-8') >>> my_string = "Hi

u'Hi \u2119\u01b4\u2602\u210c\xf8\u1f24' \u2119\u01b4\u2602\u210c\xf8\u1f24"
>>> type(my_string)
$ python2 -c "import sys; sys.getdefaultencoding()" <class 'str'>
ascii
>>> my_bytes = b"Hello World"
$ python3 -c "import sys; sys.getdefaultencoding()" >>> type(my_bytes)
utf-8 <class 'bytes'>
Language Reference (continued…)

Reading files (py3) pi_utf8.txt

>>> open('hello.txt', 'r').read() r = 10

'Hello, world!\n' π = math.pi
area = π * r ** 2
>>> open('hello.txt', 'rb').read()
b'Hello, world!\n'

>>> open('pi_utf8.txt', 'r').read()

'r = 10\nπ = math.pi\narea = π * r ** 2\n'

>>> open('pi_utf8.txt', 'rb').read()

b'r = 10\n\xcf\x80 = math.pi\narea = \xcf\x80 * r ** 2\n'

>>> open('pi_utf8.txt', 'r', encoding=locale.getpreferredencoding()).read()

'r = 10\nπ = math.pi\narea = π * r ** 2\n'
>> print('\N{GREEK SMALL LETTER PI}'
>>> open('pi_utf8.txt', 'r', encoding='utf-8').read()
'r = 10\nπ = math.pi\narea = π * r ** 2\n' π

>>> open('pi_utf8.txt', 'r', encoding='cp1252').read()

>>> open('pi_utf8.txt', 'r', encoding='windows-1252').read()
'r = 10\nπ = math.pi\narea = π * r ** 2\n'

from encodings.aliases import aliases

print(sorted(aliases.keys())
Language Reference (continued…)

Integer Literals Imaginary Literals

longinteger ::= integer ("l" | "L") *py2 imagnumber ::= (floatnumber | intpart) ("j" | "J")
integer ::= decimalinteger | octinteger
| hexinteger | bininteger
decimalinteger ::= nonzerodigit digit* | "0"
octinteger ::= "0" ("o" | "O") octdigit+ | "0" octdigit+
hexinteger ::= "0" ("x" | "X") hexdigit+
bininteger ::= "0" ("b" | "B") bindigit+ # int
nonzerodigit ::= "1"..."9" 2147483647 0o177 0b100110111
octdigit ::= "0"..."7" 3 0o377 0xdeadbeef
bindigit ::= "0" | "1" 79228162514264337593543950336
hexdigit ::= digit | "a"..."f" | "A"..."F"
# float
3.14 10. .001
Floating-point Literals 1e100 3.14e-10 0e0

floatnumber ::= pointfloat | exponentfloat # complex

pointfloat ::= [intpart] fraction | intpart "." 3.14j 10.j 10j
exponentfloat ::= (intpart | pointfloat) exponent .001j 1e100j 3.14e-10j
intpart ::= digit+
fraction ::= "." digit+
exponent ::= ("e" | "E") ["+" | "-"] digit+
Language Reference (continued…)
Expressions

class int() methods (class long)

● 'bit_length', 'conjugate', 'denominator', 'from_bytes', 'imag', 'numerator', 'real', 'to_bytes'

class float() methods

● 'as_integer_ratio', 'fromhex', 'hex', 'is_integer'

class complex() methods

● 'conjugate', 'imag', 'real'

__builtin__ math functions

● 'abs', 'cmp', 'divmod', 'max', 'min', 'pow', 'round', 'sum'
● 'bin', 'hex', 'oct',

math module exports

● 'e', 'pi'
● 'acos', 'acosh', 'asin', 'asinh', 'atan', 'atan2', 'atanh', 'cos', 'cosh', 'degrees', 'radians',, 'sin', 'sinh', 'tan', 'tanh'
● 'ceil', 'copysign', 'erf', 'erfc', 'exp', 'expm1', 'fabs', 'factorial', 'floor', 'fmod', 'frexp', 'fsum', 'gamma', 'hypot', 'isfinite', 'isinf',
'isnan', 'ldexp', 'lgamma', 'log', 'log10', 'log1p', 'log2', 'modf', 'pow', 'radians', 'sqrt', 'trunc'

*py2
*py3
Language Reference (continued…)
Expressions

● Arithmetic conversions
number operator complex → complex
number operator float → float
integer operator integer → integer

● Atoms
atom ::= identifier | literal | enclosure
enclosure ::= parenth_form | list_display | dict_display | set_display | generator_expression | yield_atom

○ Identifiers (case sensitive)

identifier ::= (letter|"_") (letter | digit | "_")*
letter ::= lowercase | uppercase
lowercase ::= "a"..."z"
uppercase ::= "A"..."Z"
digit ::= "0"..."9"

○ Literals
literal ::= stringliteral | bytesliteral | integer | floatnumber | imagnumber

○ Parenthesized forms
parenth_form ::= "(" [expression_list] ")"
Language Reference (continued…)
Expressions (continued…)

● Atoms (continued)
atom ::= identifier | literal | enclosure
enclosure ::= parenth_form | list_display | dict_display | set_display | generator_expression | yield_atom

○ Displays (comprehension) for lists, sets and dictionaries

comprehension ::= expression comp_for
comp_for ::= "for" target_list "in" or_test [comp_iter]
comp_iter ::= comp_for | comp_if
comp_if ::= "if" expression_nocond [comp_iter]

○ List display (comprehension)

list_display ::= "[" [expression_list | comprehension] "]"

○ Dictionary displays (comprehension)

dict_display ::= "{" [key_datum_list | dict_comprehension] "}"
key_datum_list ::= key_datum ("," key_datum)* [","]
key_datum ::= expression ":" expression
dict_comprehension ::= expression ":" expression comp_for

○ Set display (comprehension)

set_display ::= "{" (expression_list | comprehension) "}"
Language Reference (continued…)
Expressions (continued…)

● Atoms (continued)

○ String conversion (*py2)

string_conversion ::= "`" expression_list "`" → repr()

○ Generator expressions
generator_expression ::= "(" expression comp_for ")"
■ Generator-iterator methods
generator.__next__()
generator.send()
generator.throw(type[, value[, traceback]])
generator.close()

○ Yield expressions
yield_atom ::= "(" yield_expression ")"
yield_expression ::= "yield" [expression_list | "from" expression]
Language Reference (continued…)
Expressions (continued…)
# making a list the hard way # set comprehension
squares = [] primes = {x for x in range(2, 101) if all(x % y for y in range(2, min(x, 11)))}
for x in range(10):
squares.append(x**2) # making a set the hard way
prime_pairs = set()
# list comprehension for x in primes:
squares = [x**2 for x in range(10)] if x + 2 in primes:
prime_pairs.add((x, x + 2))
# dictionary comprehension
{n: 2**n for n in range(11)} # set comprehension
prime_pairs = {(x, x + 2) for x in primes if x + 2 in primes}
# making a set the hard way
primes = set() # generator using tuple () notation
for x in range(2, 101): >>> g = (x for x in range(3))
for y in range(2, min(x, 11)): >>> g.next()
if not x % y: 0
break >>> g.next()
else: 1
primes.add(x) >>> g.next()
2
# nested the hardway >>> g.next()
matrix = [[0,1,2,3], [4,5,6,7], [8,9,10,11]] Traceback (most recent call last):
flat = [] File "<stdin>", line 1, in <module>
for row in matrix:
StopIteration
for col in row:
* unpacks a list or tuple into positional arguments
flat.append(col)
** unpacks a dictionary into keyword arguments
# nested
flat = [c for row in matrix for c in row] # [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]
transposed = [[row[c] for row in matrix] for c in range(len(row))] → list(zip(*matrix)) # [(0,4,8), (1,5,9), (2,6,10), (3,7,11)]
Language Reference (continued…)
Expressions (continued…)
# iterator # generator using function and yield
l = [1, 2, 3, 4] def fibonacci(n):
it = iter(l) a, b, counter = 0, 1, 0
while True:
# Using for loop if counter > n:
for x in it: return
print(x, end=' ') yield a
a, b = b, a + b
# Using while loop and next() counter += 1
while True:
try: # generator is an iterator object
print(next(it)) f = fibonacci(5)
except StopIteration:
break while True:
try: # Using for loop
print (next(f), end=" ") for x in f:
except StopIteration: print(x, end=' ')
break

http://www.tutorialspoint.com/python3/python_loops.htm
Language Reference (continued…)
Expressions (continued…)

● Primaries
primary ::= atom | attributeref | subscription | slicing | call
○ Attribute references
attributeref ::= primary "." identifier → module, list, instance attribute
○ Subscriptions
subscription ::= primary "[" expression_list "]" → string, tuple, list, dictionary
○ Slicing → string, tuple, list
slicing ::= primary "[" slice_list "]"
slice_list ::= slice_item ("," slice_item)* [","]
slice_item ::= expression | proper_slice
proper_slice ::= [lower_bound] ":" [upper_bound] [ ":" [stride] ]
lower_bound ::= expression
upper_bound ::= expression
stride ::= expression s[start:end] # start to end-1
s[start:] # start to end
s[:end] # beginning to end-1
s[:] # a copy of sequence
s[::-1] # reverse sequence

s[-1] # last item

s[-2:] # last two items
s[:-2] # everything except the last two items
Language Reference (continued…)
Expressions (continued…)

● Primaries (continued)
○ Calls → callable object (function, class, method, ...)
call ::= primary "(" [argument_list [","] | comprehension] ")"
argument_list ::= positional_arguments ["," keyword_arguments]
["," "*" expression] ["," keyword_arguments]
["," "**" expression]
| keyword_arguments ["," "*" expression]
["," keyword_arguments] ["," "**" expression]
| "*" expression ["," keyword_arguments] ["," "**" expression]
| "**" expression
positional_arguments ::= expression ("," expression)*
>>> def f(a, b=1, *c, **d):
keyword_arguments ::= keyword_item ("," keyword_item)*
... print(a, b, c, d)
keyword_item ::= identifier "=" expression >>> f(1)
(1, 1, (), {})
>>> f(1, 3)
Argument (1, 3, (), {})
1. Positional arguments >>> f(1, 3, 5)
2. Positional arguments with default value (1, 3, (5,), {})
>>> f(1, 3, 5, 6)
3. Non-keyword args → tuple >>> f(1, 3, 5, 6, x=5)
4. Keyword args → dictionary (1, 3, (5, 6), {'x': 5})
>>> f(1, 3, 5, 6, x=5, y=6)
(1, 3, (5, 6), {'y': 6, 'x': 5})
Language Reference (continued…)
Expressions (continued…)

● Primaries (continued)
○ Await expression → (*py3)
await ::= ["await"] primary
○ Power operator (pow function)
power ::= await ["**" u_expr]
○ Unary arithmetic and bitwise operations
u_expr ::= power | "-" u_expr | "+" u_expr | "~" u_expr
○ Binary arithmetic operations
m_expr ::= u_expr
| m_expr "*" u_expr | m_expr "@" m_expr
| m_expr "//" u_expr | m_expr "/" u_expr (*py3: The / (division) and // (floor division))
| m_expr "%" u_expr
a_expr ::= m_expr | a_expr "+" m_expr | a_expr "-" m_expr
○ Shifting operations
shift_expr ::= a_expr | shift_expr ( "<<" | ">>" ) a_expr
○ Binary bitwise operations
and_expr ::= shift_expr | and_expr "&" shift_expr
xor_expr ::= and_expr | xor_expr "^" and_expr
or_expr ::= xor_expr | or_expr "|" xor_expr
Language Reference (continued…)
Expressions (continued…)

● Primaries (continued)
○ Comparisons
comparison ::= or_expr ( comp_operator or_expr )*
comp_operator ::= "<" | ">" | "==" | ">=" | "<=" | "!=" | "is" ["not"] | ["not"] "in"
■ Value comparison "<" | ">" | "==" | ">=" | "<=" | "!="
■ Membership test operations "in" | "not in"
■ Identity comparison "is" | "is not"
○ Boolean operations
or_test ::= and_test | or_test "or" and_test
and_test ::= not_test | and_test "and" not_test
not_test ::= comparison | "not" not_test
○ Conditional expressions
conditional_expression ::= or_test ["if" or_test "else" expression]
expression ::= conditional_expression | lambda_expr
expression_nocond ::= or_test | lambda_expr_nocond
○ Lambdas
lambda_expr ::= "lambda" [parameter_list]: expression
lambda_expr_nocond ::= "lambda" [parameter_list]: expression_nocond
○ Expression lists
expression_list ::= expression ( "," expression )* [","]
Operators

Assignment: <var_name> = <value> - a, b = b, a # swap Comparison:

Arithmetic: Operator Operation

Operator Operation ==, (!=, <>) Equal/Not equal

+, -, +=, -= Addition/Subtraction
<, > Less than/Greater than
*, /, *=, /= Multiplication/Division
<=, >= Less than or equal/Greater than or equal
%, //, %=, //= Modulus/Quotient (integer division)

**, **= Exponent Bitwise:

Operator Operation
Boolean: and, or, not

Sequences (string, list, tuple): <<, >> Shift left/Shift right

Operator Operation ~ not

+ Concatenation &, |, ^ and, or, xor

&=, |=, ^=
* Replication
Operator Precedence

Operator Description

lambda Lambda expression

if – else Conditional expression → (a = b if cond else c)
or Boolean OR
and Boolean AND
not x Boolean NOT
in, not in, is, is not, <, <=, >, >=, !=, == Comparisons, including membership tests and identity tests
| Bitwise OR
^ Bitwise XOR
& Bitwise AND
<<, >> Shifts
+, - Addition and subtraction
*, @, /, //, % Multiplication, matrix multiplication division, remainder
+x, -x, ~x Positive, negative, bitwise NOT
** Exponentiation
await x Await expression (*py3)
x[index], x[index:index], x(arguments...), x.attribute Subscription, slicing, call, attribute reference
(expressions...), [expressions...], {key: value...}, {expressions...} Binding or tuple display, list display, dictionary display, set display
Language Reference (continued…)
Statements
simple_stmt ::= expression_stmt
| assert_stmt
| assignment_stmt
| augmented_assignment_stmt
| pass_stmt
| del_stmt
| return_stmt
| yield_stmt
| raise_stmt
| break_stmt
| continue_stmt
| import_stmt
| global_stmt
| nonlocal_stmt (*py3)
Language Reference (continued…)
Statements (continued…)

Expression Statement
expression_stmt ::= expression_list

Assignment statements
assignment_stmt ::= (target_list "=")+ (expression_list | yield_expression)
target_list ::= target ("," target)* [","]
target ::= identifier
| "(" target_list ")"
| "[" target_list "]"
| attributeref
| subscription
| slicing
| "*" target

x = [0, 1]
i = 0
i, x[i] = 1, 2 # i is updated, then x[i] is updated
a, b = b, a # swap a and b

Augmented assignment statements

augmented_assignment_stmt ::= augtarget augop (expression_list | yield_expression)
augtarget ::= identifier | attributeref | subscription | slicing
augop ::= "+=" | "-=" | "*=" | "@=" | "/=" | "//=" | "%=" | "**="
| ">>=" | "<<=" | "&=" | "^=" | "|="
x += 1
Language Reference (continued…)
Statements (continued…)

assert → (can use affirm for better message output)

assert_stmt ::= "assert" expression ["," expression]

assert expression → if __debug__: if not expression: raise AssertionError

assert expression1, expression2 → if __debug__: if not expression1: raise AssertionError(expression2)
pass
pass_stmt ::= "pass"

if a:
pass
else:
x()

def f(): pass

class C(object): pass

while True:
pass # Busy-wait for keyboard interrupt (Ctrl+C)
del
del_stmt ::= "del" target_list

d = {'a': 1, 'b': 2}
del d
Language Reference (continued…)
Statements (continued…)

print → *py2
print_stmt ::= "print" ([expression ("," expression)* [","]] | ">>" expression [("," expression)+ [","]])

import sys
print >> sys.stderr, sys.version → sys.stderr.write(sys.version + '\n')
return
return_stmt ::= "return" [expression_list]

def triple(x): return x * 3

yield
yield_stmt ::= yield_expression

def mygen(n):
l = range(n)
for i in l:
yield i * 2 raise IOError, "file error" # accepted in Python 2
raise IOError("file error") # accepted in Python 2
for i in mygen(4): raise IOError, "file error" # syntax error in Python 3
raise IOError("file error") # recommended in Python 3
print i
raise
raise_stmt ::= "raise" [expression ["," expression ["," expression]]]

raise SystemExit
raise RuntimeError('Something bad happened')
Language Reference (continued…)
Statements (continued…)

break
break_stmt ::= "break"

for n in range(2, 10):

for x in range(2, n):
if n % x == 0:
print n, 'equals', x, '*', n/x
break
else:
# loop fell through without finding a factor
print n, 'is a prime number'

continue
continue_stmt ::= "continue"

for num in range(2, 10):

if num % 2 == 0:
print "Found an even number", num
continue
print "Found a number", num
Language Reference (continued…)
Statements (continued…)

import
import_stmt ::= "import" module ["as" name] ( "," module ["as" name] )*
| "from" relative_module "import" identifier ["as" name]
( "," identifier ["as" name] )*
| "from" relative_module "import" "(" identifier ["as" name]
( "," identifier ["as" name] )* [","] ")"
| "from" module "import" "*"

import foo # foo imported and bound locally

import foo.bar.baz # foo.bar.baz imported, foo bound locally
import foo.bar.baz as fbb # foo.bar.baz imported and bound as fbb
from foo.bar import baz # foo.bar.baz imported and bound as baz
from foo import attr # foo imported and foo.attr bound as attr

future
future_statement ::= "from" "__future__" "import" feature ["as" name]
("," feature ["as" name])*
| "from" "__future__" "import" "(" feature ["as" name]
("," feature ["as" name])* [","] ")"
feature ::= identifier
name ::= identifier

from __future__ import (absolute_import, division, print_function, unicode_literals)

Language Reference (continued…)
Statements (continued…)

global
global_stmt ::= "global" identifier ("," identifier)*

global a, b, c

nonlocal → *py3
nonlocal_stmt ::= "nonlocal" identifier ("," identifier)*

def f(a):
z = 1
def f1(b):
nonlocal z
return b + z
return a + f1(a)

exec → *py2 built-in function in py3

exec_stmt ::= "exec" or_expr ["in" expression ["," expression]]

>>> exec('a = 10; print a')

10
Language Reference (continued…)
Compound Statements
compound_stmt ::= if_stmt
| while_stmt
| for_stmt
| try_stmt
| with_stmt
| funcdef
| classdef
| decorated
| async_with_stmt *py3
| async_for_stmt *py3
| async_funcdef *py3

suite ::= stmt_list NEWLINE | NEWLINE INDENT statement+ DEDENT

statement ::= stmt_list NEWLINE | compound_stmt
stmt_list ::= simple_stmt (";" simple_stmt)* [";"]
Language Reference (continued…)
Compound Statements (continued…)

if for
if_stmt ::= "if" expression ":" suite for_stmt ::= "for" target_list "in" expression_list ":"
( "elif" expression ":" suite )* suite
[ "else" ":" suite ] [ "else" ":" suite ]

if 0 < a < 10:

for i in iterable:
pass
# do something
elif b < 20:
else:
pass
# do something at the end of the loop if no break
elif c:
pass

while
while_stmt ::= "while" expression ":" suite
["else" ":" suite]

while cond:
# do something
else:
# do something else if cond is false
Language Reference (continued…)
Compound Statements (continued…)

try
try_stmt ::= try1_stmt | try2_stmt
try1_stmt ::= "try" ":" suite
("except" [expression [("as" | ",") identifier]] ":" suite)+
["else" ":" suite]
["finally" ":" suite]
try2_stmt ::= "try" ":" suite
"finally" ":" suite
except Myerror, err: # Python 2
except Myerror as err: # Python 3
try:
# do something except (RuntimeError, TypeError, NameError):
except Exception as e
# do some error handling # Reads file in one shot into a string
else: with open('filename.txt') as f:
# do something at the end if exceptions don’t occur data = f.read()
finally:
# cleanup code on the way out # Reads all lines and puts them in a list
with open('filename.txt') as f:
lines = f.readlines()
with → context managers (methods: __enter__/__exit__)
with_stmt ::= "with" with_item ("," with_item)* ":" suite # Reads file line by line
with_item ::= expression ["as" target] with open('filename.txt') as f:
for line in f:
print(line)
Language Reference (continued…)
Compound Statements (continued…)

Timer context manager

#!/usr/bin/env python
import datetime
import sys
import time

# On Windows, the best timer is time.clock

# On other platforms the best timer is time.time
default_timer = time.clock if sys.platform == 'win32' else time.time

class Timer(object):

def __init__(self, verbose=False):

self.verbose = verbose

def __enter__(self):
# self.start = time.clock()
self.start = time.time()
self.t1 = datetime.datetime.now()
return self

def __exit__(self, *args):

# self.end = time.clock()
self.end = time.time()
self.t2 = datetime.datetime.now()
if self.verbose:
print('Elapsed: {}.'.format(self.t2 - self.t1, grouping=True))

if __name__ == '__main__':
with Timer(verbose=True) as t:
with open(sys.argv[1], 'r') as f:
f.read()
Language Reference (continued…)

function # mytimeit.py
import time

decorated ::= decorators (classdef | funcdef) def timeit(method):

decorators ::= decorator+ def timed(*args, **kw):
decorator ::= "@" dotted_name ["(" [argument_list [","]] ")"] NEWLINE ts = time.time()
funcdef ::= "def" funcname "(" [parameter_list] ")" ":" suite result = method(*args, **kw)
te = time.time()
dotted_name ::= identifier ("." identifier)*
parameter_list ::= (defparameter ",")* print('%r (%r, %r) %2.2f sec' % (
( "*" identifier ["," "**" identifier] method.__name__,
| "**" identifier args, kw, te-ts
)
| defparameter [","] ) )
defparameter ::= parameter ["=" expression] return result
sublist ::= parameter ("," parameter)* [","]
return timed
parameter ::= identifier | "(" sublist ")"
funcname ::= identifier # test.py
import sys
import mytimeit
# Decorated function
@f1(arg) @mytimeit.timeit
@f2 def read_file(filename):
def func(): pass with open(filename, 'r') as f:
return f.read()
# Equivalent to if __name__ == '__main__':
def func(): pass read_file(sys.argv[1])
func = f1(arg)(f2(func))
Language Reference (continued…)

class

classdef ::= "class" classname [inheritance] ":" suite

inheritance ::= "(" [expression_list] ")"
classname ::= identifier

● Class creation can be customized with metaclasses

● Classes can be decorated like functions
● Classes inherit from object by default (py3), or should be explicitly inherited from object (py2)
Language Reference (continued…)

Coroutines (*py3) (used with asyncio, ...)

● Coroutines is a more generalized form of subroutines. Subroutines are entered at one point and exited at another point.
Coroutines can be entered, exited, and resumed at many different points.

async_funcdef ::= [decorators] "async" "def" funcname "(" [parameter_list] ")" ["->" expression] ":" suite
async_for_stmt ::= "async" for_stmt
async_with_stmt ::= "async" with_stmt

import asyncio
import datetime

async def display_date(loop):

end_time = loop.time() + 5.0
while True:
print(datetime.datetime.now())
if (loop.time() + 1.0) >= end_time:
break
await asyncio.sleep(1)

loop = asyncio.get_event_loop() 2016-06-14 10:34:15.636135

# Blocking call which returns when 2016-06-14 10:34:16.637486
# the display_date() coroutine is done 2016-06-14 10:34:17.638899
loop.run_until_complete(display_date(loop)) 2016-06-14 10:34:18.640318
2016-06-14 10:34:19.641704
loop.close()
Python File I/O
File Object and its Methods

__builtins__
● raw_input()
str = raw_input("Enter a string: ")
print("str: %s" % str)
● input()
value = input("Enter new value: ")
print("value: {}".format(value))
Builtin function open():
<file object> = open(file_name [, access_mode][, buffering])
access_mode :: ( 'r' | 'rb' | 'r+' | 'rb+' | 'w' | 'wb' | 'w+' | 'wb+' | 'a' | 'ab' | 'a+' | 'ab+' )
Attributes:
● closed, encoding, errors, line_buffering, mode, name
Methods:
● close(), flush(), isatty(), read(), readline(), readlines(), seek(), tell(), truncate(), write(), write_lines()
File Object and its Methods (continued…)

● The type of file object returned by the open() function depends on the mode.
○ When open() is used to open a file in a text mode ('w', 'r', 'wt', 'rt', etc.), it returns a subclass of io.TextIOBase
(specifically io.TextIOWrapper).
○ When used to open a file in a binary mode with buffering, the returned class is a subclass of io.BufferedIOBase.
The exact class varies: in read binary mode,
■ returns an io.BufferedReader; in write binary and append binary modes,
■ returns an io.BufferedWriter, and in read/write mode
■ returns an io.BufferedRandom.
■ When buffering is disabled, the raw stream, a subclass of io.RawIOBase, io.FileIO, is returned.
Useful File-Related modules/methods

os module functions:
● os.access(path, mode), os.chdir(path), os.chflags(path, flags), os.chmod(path, mode)
● os.chown(path, uid, gid), os.chroot(path), os.close(fd), os.closerange(fd_low, fd_high), os.dup(fd)
● os.dup2(fd, fd2), os.fchdir(fd), os.fchmod(fd, mode), os.fchown(fd, uid, gid), os.fdatasync(fd)
● os.fdopen(fd[, mode[, bufsize]]), os.fpathconf(fd, name), os.fstat(fd), os.fstatvfs(fd), os.fsync(fd)
● os.ftruncate(fd, length), os.getcwd(), os.getcwdu(), os.isatty(fd), os.lchflags(path, flags)
● os.lchmod(path, mode), os.lchown(path, uid, gid), os.link(src, dst), os.listdir(path)
● os.lseek(fd, pos, how), os.lstat(path), os.major(device), os.makedev(major, minor)
● os.makedirs(path[, mode]), os.minor(device), os.mkdir(path[, mode]), os.mkfifo(path[, mode])
● os.mknod(filename[, mode=0600, device]), os.open(file, flags[, mode]), os.openpty()
● os.pathconf(path, name), os.pipe(), os.popen(command[, mode[, bufsize]]), os.read(fd, n)
● os.readlink(path), os.remove(path), os.removedirs(path), os.rename(src, dst), os.renames(old, new)
● os.rmdir(path), os.stat(path), os.stat_float_times([newvalue]), os.statvfs(path), os.symlink(src, dst)
● os.tcgetpgrp(fd), os.tcsetpgrp(fd, pg), os.tempnam([dir[, prefix]]), os.tmpfile(), os.tmpnam()
● os.ttyname(fd), os.unlink(path), os.utime(path, times)
● os.walk(top[, topdown=True[, onerror=None[, followlinks=False]]]), os.write(fd, str)

os.path module functions

● os.path.join(), os.path.abspath(), os.path.isabs(), os.path.relpath(),
● os.path.basename(), os.path.dirname(), os.path.getsize()
● os.path.exists(), os.path.isfile(), os.path.isdir()
File I/O

$ python $ python
# Open file to write to $ cat f.txt # read a file in whole, return string
>>> fo = open('f.txt', 'w') Line 1 >>> with open('f.txt', 'r') as f:
>>> fo.write('Line 1\n') Line 2 ... content = f.read()
>>> fo.write('Line 2\n') Line 3
>>> fo.write('Line 3\n') >>> content
>>> fo.close() 'Line 1\nLine 2\nLine 3\n'
>>> print(c, end='')
>>> f = open('f.txt', 'r') Line 1
>>> f.name Line 2
'f.txt' Line 3
>>> f.closed
False # read a file in whole, return list of lines
>>> f.mode >>> with open('f.txt', 'r') as f:
'r' ... content = f.readlines()
>>> f.tell() ...
0 >>> content
>>> f.isatty() ['Line 1\n', 'Line 2\n', 'Line 3\n']
False
c = f.read(10)
>>> c
'Line 1\nLin'
>>> f.tell()
10
>>> f.close()
>>> <CTRL+D>
Python OOP and Class Type
New Style vs Old Style Classes

New style vs old style classes

● Up to Python 2.1 the concept of class was unrelated to the concept of type, and old-style classes were
the only flavor available. For an old-style class, the statement x.__class__ provides the class of x, but
type(x) is always <type 'instance'>. This reflects the fact that all old-style instances, independent of their
class, are implemented with a single built-in type, called instance.
○ e.g.: class A:
● New-style classes were introduced in Python 2.2 to unify the concepts of class and type. A new-style
class is simply a user-defined type, no more, no less. If x is an instance of a new-style class, then
type(x) is typically the same as x.__class__ (although this is not guaranteed – a new-style class instance
is permitted to override the value returned for x.__class__).
○ e.g.: class A(object): # *in py3 this is default
Python Class

Class
● class attribute (class variable)
● class method (static method)
Instance
● instance attribute (member variable)
● instance method

Class magic method Description Called when

__init__(self [,args...]) Constructor (with any optional arguments) obj = className(args)
__del__(self) Destructor, deletes an object del obj
__repr__(self) Evaluatable string representation repr(obj)
__str__(self) Printable string representation str(obj)
__cmp__(self, x) Object comparison cmp(obj, x)
Python Class Inheritance (specialization - to be)

Car
Delegation
● Implement new features
● Modify parent features
# py2 # py3 SportsCar Truck
class Car(object): class Car:
def __init__(self): pass def __init__(self): pass
def method(self): pass def method(self): pass

class SportsCar(Car): class SportsCar(Car):

def __init__(self): pass def __init__(self): pass

def method(self): def method(self):

super(SportsCar, self).method() super().method()

class Truck(Car): class Truck(Car):

def __init__(self): pass def __init__(self): pass

def method(self): def method(self):

super(SportsCar, self).method() super().method()

Class

Attributes

Operations
Python Class Composition (ownership - to have)

Car
Composition
(is part of) 1 1 (is part of)
(has) 4 1 (has)
# py2 # py3 Wheel Engine
class Wheel(object): class Wheel:
def __init__(self): pass def __init__(self): pass

class Engine(object): class Engine:

def __init__(self): pass def __init__(self): pass

class Car(object): class Car:

def __init__(self): def __init__(self):
for n in range(4): for n in range(4):
self.wheels[n] = Wheel() self.wheels[n] = Wheel()
self.engine = Engine() self.engine = Engine()
Object Count and Class Properties

# c.py # c.py
class C(object): class C(object): $ python c.py
# Class member variable # Class member variable 'C' object has no attribute '__obj_count'
_obj_count = 0 __obj_count = 0 type object 'C' has no attribute '__obj_count'
3
def __init__(self): def __init__(self):
# self.__class__._obj_count += 1 # self.__class__._obj_count += 1
C._obj_count += 1 C.__obj_count += 1

@property @property
def obj_count(self): def obj_count(self):
return self._obj_count return self.__obj_count

c1 = C() c1 = C()
print(c1._obj_count) try:
print(c1.__obj_count)
c2 = C() except Exception as e:
print(C._obj_count) print(e)

c3 = C() c2 = C()
print(c3.obj_count) try:
print(C.__obj_count)
except Exception as e:
$ python c.py print(e)
1
2 c3 = C()
3 print(c3.obj_count)
Special Methods

Basic customization Customizing attribute access

● object.__new__(cls[, ...]) ● object.__getattr__(self, name)
● object.__init__(self[, ...]) ● object.__getattribute__(self, name) # * new style classes or py3
● object.__del__(self) ● object.__setattr__(self, name, value)
● object.__repr__(self) ● object.__delattr__(self, name)
● object.__str__(self) ● object.__dir__(self)
● object.__bytes__(self) Implementing Descriptors
● object.__format__(self, format_spec)
● object.__get__(self, instance, owner)
● object.__lt__(self, other)
● object.__set__(self, instance, value)
● object.__le__(self, other)
● object.__delete__(self, instance)
● object.__eq__(self, other)
● object.__ne__(self, other) __slots__
● object.__gt__(self, other) ● __slots__
● object.__ge__(self, other) Customizing class creation
● object.__cmp__(self, other) ● __metaclass__
● object.__rcmp__(self, other)
● object.__hash__(self)
○ Metaclasses alter or monitor class creation process
● object.__bool__(self) Customizing instance and subclass checks
● object.__nonzero__(self) ● class.__instancecheck__(self, instance)
● object.__unicode__(self) ● class.__subclasscheck__(self, subclass)
Emulating callable objects
● object.__call__(self[, args...])

*py2
*py3
Special Methods (continued)

Emulating container types ● object.__xor__(self, other) ● object.__ipow__(self, other[, modulo])

● object.__len__(self) ● object.__or__(self, other) ● object.__ilshift__(self, other)
● object.__length_hint__(self) # py3 ● object.__radd__(self, other) ● object.__irshift__(self, other)
● object.__getitem__(self, key) ● object.__rsub__(self, other) ● object.__iand__(self, other)
● object.__missing__(self, key) ● object.__rmul__(self, other) ● object.__ixor__(self, other)
● object.__setitem__(self, key, value) ● object.__rmatmul__(self, other) ● object.__ior__(self, other)
● object.__delitem__(self, key) ● object.__rtruediv__(self, other) ● object.__neg__(self)
● object.__iter__(self) ● object.__rfloordiv__(self, other) ● object.__pos__(self)
● object.__reversed__(self) ● object.__rmod__(self, other) ● object.__abs__(self)
● object.__contains__(self, item) ● object.__rdivmod__(self, other) ● object.__invert__(self)
● object.__rpow__(self, other) ● object.__complex__(self)
Emulating numeric types ● object.__rlshift__(self, other) ● object.__int__(self)
● object.__add__(self, other) ● object.__rrshift__(self, other) ● object.__float__(self)
● object.__sub__(self, other) ● object.__rand__(self, other) ● object.__round__(self[, n])
● object.__mul__(self, other) ● object.__rxor__(self, other) ● object.__index__(self)
● object.__matmul__(self, other) ● object.__ror__(self, other)
● object.__truediv__(self, other) ● object.__iadd__(self, other)
● object.__floordiv__(self, other) ● object.__isub__(self, other)
● object.__mod__(self, other) ● object.__imul__(self, other)
● object.__divmod__(self, other) ● object.__imatmul__(self, other)
● object.__pow__(self, other[, modulo]) ● object.__itruediv__(self, other)
● object.__lshift__(self, other) ● object.__ifloordiv__(self, other)
● object.__rshift__(self, other) ● object.__imod__(self, other)
● object.__and__(self, other)

*py2
*py3
Special Methods (continued)

With Statement Context Managers

● object.__enter__(self)
● object.__exit__(self, exc_type, exc_value, traceback)
Coroutines
● Awaitable Objects
○ object.__await__(self)
● Coroutine Objects
○ coroutine.send(value)
○ coroutine.throw(type[, value[, traceback]])
○ coroutine.close()
Asynchronous Iterators
● object.__aiter__(self)
● object.__anext__(self)
Asynchronous Context Managers
● object.__aenter__(self)
● object.__aexit__(self, exc_type, exc_value, traceback)

*py2
*py3
Class Variables and Class Methods

# #!/usr/bin/env python $ python c.py

5
from __future__ import print_function
DEBUG : I am static - 10
# Class variables and methods INFO : I am static - 20
class C(object): 30
# Class member variable
_static_var = 'I am static'
_static_var1 = 10

def __init__(self, x):

self.x = x

@classmethod
def static_method(cls, msg, y):
print('{} : {} - {}'.format(msg, cls._static_var, cls._static_var1))
cls._static_var1 = y

if __name__ == '__main__':
c = C(5)
print(c.x)
c.static_method('DEBUG', 20)

C.static_method('INFO', 30)
print(C._static_var1)
Private Class Members

#!/usr/bin/env python $ python c.py

1
from __future__ import print_function
'C' object has no attribute '__private'
# Private member variables 2
class C(object):

def __init__(self, x, y):

"""Constructor"""
self._protected = x
self.__private = y

if __name__ == '__main__':
c = C(1, 2)

print(c._protected)

try:
print(c.__private)
except Exception as e:
print(e)

print(c._C__private)
Python Instance Properties (managed attribute)

# Method 1 # Method 3
# class property([fget[, fset[, fdel[, doc]]]]) class C(object):
class C(object): def __init__(self): self._x = None
def __init__(self): self._x = None
def x(self):
def getx(self): return self._x doc = "I'm the 'x' property."
def fget(self): return self._x
def setx(self, value): self._x = value def fset(self, value): self._x = value
def fdel(self): del self._x
def delx(self): del self._x
return locals()
x = property(getx, setx, delx, "I'm the 'x' property.")
x = property(**x())
# Method 2
class C(object):
def __init__(self): self._x = None

@property
def x(self):
"""I'm the 'x' property."""
return self._x

@x.setter
def x(self, value): self._x = value

@x.deleter
def x(self): del self._x ;
Python Instance Properties (managed attribute) - continued...

>>> c = C() # Method 2 and 3

>>> c.x = [1, 2, 3, 4] Help on class C:
>>> c.x
[1, 2, 3, 4] class C(__builtin__.object)
>>> help(C) | Methods defined here:
# Method 1 |
Help on class C: | __init__(self)
|
class C(__builtin__.object) | -------------------------------------
| Methods defined here: | Data descriptors defined here:
| |
| __init__(self) | __dict__
| | dictionary for instance variables (if defined)
| delx(self) |
| | __weakref__
| getx(self) | list of weak references to the object (if defined)
| |
| setx(self, value) | x
| | I'm the 'x' property.
| --------------------------------------------------
| Data descriptors defined here:
|
| __dict__
| dictionary for instance variables (if defined)
|
| __weakref__
| list of weak references to the object (if defined)
|
| x
| I'm the 'x' property.
Abstract Base Classes and Polymorphism

from __future__ import print_function $ python c.py

import abc
Cat is a subclass of Animal: True
class Animal(object): # py3: class Animal(abc.ABC): Cat() is an instance of Animal: True
__metaclass__ = abc.ABCMeta # py2
Can't instantiate abstract class Animal
@abc.abstractmethod
def sound(self): pass
with abstract methods sound

def action(self): Miauuu!

raise NotImplementedError() Sleep...
Woofff!
class Cat(Animal):
def sound(self): print("Miauuu!")
def action(self): print('Sleep...') Miauuu!
Woofff!
class Dog(Animal):
def sound(self): print("Woofff!")
Traceback (most recent call last):
if __name__ == '__main__': File "d.py", line 38, in <module>
print('Cat is a subclass of Animal:', issubclass(Cat, Animal)) d.action()
print('Cat() is an instance of Animal:', isinstance(Cat(), Animal), '\n') File "d.py", line 15, in action
try:
raise NotImplementedError()
a = Animal() NotImplementedError
except Exception as e:
print(e, '\n')

c = Cat() ; c.sound() ; c.action()

d = Dog() ; d.sound() ; print()

animals = (c, d); for a in animals: a.sound()

print() ; d.action()
Object Attribute Access

# Ask for permission

if hasattr(obj, method):
# do something
else:
# do something else

# Ask for forgiveness

try:
obj.method()
except AttributeError:
# oops! sorry.
Function defaults, *args and **kwargs

from __future__ import print_function $ python prog.py ~~~~~~~~~~~~~~~~~~~~

~~~~~~~~~~~~~~~~~~~~ a = 0
# Class variables and methods a = 10 b = 1
def f(a, b=1, *args, **kwargs): b = 1 args = ()
print('~'*20) args = () kwargs = {'y': 12, 'x': 11, 'z': 13}
print('a =', a) kwargs = {} arg[y] = 12
print('b =', b) ~~~~~~~~~~~~~~~~~~~~ arg[x] = 11
a = 5 arg[z] = 13
print('args =', args) b = 6 ~~~~~~~~~~~~~~~~~~~~
for i, arg in enumerate(args): args = () a = 3
print(' arg[{}] = {}'.format(i, arg)) kwargs = {} b = 4
~~~~~~~~~~~~~~~~~~~~ args = (7, 8, 9)
print('kwargs =', kwargs) a = 3 arg[0] = 7
for k, v in kwargs.items(): b = 4 arg[1] = 8
print(' arg[{}] = {}'.format(k, v)) args = (7, 8, 9) arg[2] = 9
arg[0] = 7 kwargs = {'y': 12, 'x': 11, 'z': 13}
if __name__ == '__main__': arg[1] = 8 arg[y] = 12
f(10) arg[2] = 9 arg[x] = 11
f(5, 6) kwargs = {} arg[z] = 13
f(3, 4, 7, 8, 9)
f(0, x=11, y=12, z=13)
f(3, 4, 7, 8, 9, x=11, y=12, z=13)
* and **

Inside a function header:

* collects all the positional arguments in a tuple

** collects all the keyword arguments in a dictionary

def f(*args, **kwargs):

print(args)
print(kwargs)

>>> f(1, 2, 3, 4, x=1, s='hi', t=(10, 11))

(1, 2, 3, 4)
{'x': 1, 's': 'hi', 't': (10, 11)}

In a function call:

* unpacks a list or tuple into positional arguments

** unpacks a dictionary into keyword arguments

>>> lis = [1, 2, 3, 4]

>>> dic = {'x'=1, 's'='hi', 't'=(10, 11)}
>>> f(*lis, **dic) # → f(1, 2, 3, 4, x=1, s='hi', t=(10, 11))
(1, 2, 3, 4)
{'x': 1, 's': 'hi', 't': (10, 11)}
Unpacking (python3)

# Python2
a, b, c = range(3) # a = 0, b = 1, c = 2

# Python3
a, b, *c = range(5) # a = 0, b = 1, c = [2, 3, 4]
a, *b, c = range(5) # a = 0, b = [1, 2, 3], c = 4

# Use of the * and ** unpacking operators in literals, or multiple times in function calls

print(*range(3), *range(3)) # 0 1 2 0 1 2

x = [*range(3), *range(3)] # x = [0, 1, 2, 0, 1, 2]

y = {*range(3), *range(3)} # y = {0, 1, 2} (it's a set, remember!)
z = {**dict1, **dict2} # finally, syntax for dict merging!
Operator Overloading/Polymorphism

#!/usr/bin/env python def __sub__(self, other):

"""Returns the vector difference of self and other"""
from __future__ import print_function subbed = tuple(a - b for a, b in zip(self, other))
return Vector(*subbed)
import math
def dot_product(self, other):
class Vector(object): """Returns the dot product (inner product) of self and other vector"""
def __init__(self, *args): mult = tuple(a * b for a, b in zip(self, other))
"""Create a vector return Vector(*mult)
v = Vector(1, 2)"""
self.values = (0, 0) if len(args) == 0 else args def __mul__(self, other):
"""Returns the dot product of self and other if multiplied by another
def __str__(self): return str(self.values) Vector. If multiplied by an int or float, multiplies each component by other.
"""
def __repr__(self): return 'Vector' + self.__str__() if type(other) == type(self):
return self.dot_product(other)
def __len__(self): return len(self.values) elif type(other) == type(1) or type(other) == type(1.0):
product = tuple(a * other for a in self )
def __getitem__(self, key): return self.values[key] return Vector(*product)

def __setitem__(self, key, item): self.vector[key] = item if __name__ == '__main__':

v1 = Vector(1, 2, 3)
def __iter__(self): return self.values.__iter__() v2 = Vector(3, 2, 1) str(v1) : (1, 2, 3)
print('str(v1) :', v1) str(v2) : (3, 2, 1)
def __abs__(self): print('str(v2) :', v2) repr(v1) : Vector(1, 2, 3)
"""Returns the norm (length, magnitude) of the vector""" print('repr(v1) :', repr(v1)) len(v1) : 3
return math.sqrt(sum(v**2 for v in self)) print('len(v1) :', len(v1)) abs(v1) : 3.74165738677
print('abs(v1) :', abs(v1)) v1[0] : 1
def __add__(self, other): print('v1[0] :', v1[0]) v1[1] : 2
"""Returns the vector addition of self and other""" print('v1[1] :', v1[1]) v1 + v2 : (4, 4, 4)
added = tuple(a + b for a, b in zip(self, other)) print('v1 + v2 :', v1 + v2) v1 - v2 : (-2, 0, 2)
return Vector(*added) print('v1 - v2 :', v1 - v2) v1 * v2 : (3, 4, 3)
print('v1 * v2 :', v1 * v2) v1 * 2 : (2, 4, 6)
print('v1 * 2 :', v1 * 2)
Future-proofing Python (for Python 2 users only)

http://python-future.org/
http://python-future.org/compatible_idioms.html
http://pythonhosted.org/six/

# -*- coding: <encoding-name> -*-

# -*- coding: utf-8 -*-

Only if you are writing python 2 and want to be compatible with python 3:

# Future-proof for Py2 and Py3

from __future__ import (absolute_import, division, print_function,
unicode_literals)

# pylint: disable=redefined-builtin,unused-wildcard-import
from builtins import *
from future import standard_library
import six
standard_library.install_aliases()
Python Standard Library
Built-in Functions

abs(x) class frozenset() print()

all(iterable) getattr() class property()
any(iterable) globals() range()
ascii(object) → repr() in py2 hasattr() raw_input()
basestring() hash() reduce()
bin() help() reload()
class bool() hex() repr() → eval(repr(object)) == object
class bytearray() id() reversed()
class bytes() input() → eval(raw_input()) round()
callable() class int() class set()
chr() isinstance() setattr()
classmethod() → decorator issubclass() class slice()
cmp() iter() sorted()
compile() len() staticmethod()
class complex() class list() class str()
delattr() locals() sum()
class dict() long() super()
dir() map() tuple()
divmod() max() class type()
enumerate() memoryview() unichr()
eval() min() unicode()
exec() next() vars()
execfile() class object() xrange()
file() oct() zip() py2 and py3
filter() open() __import__() py2 only (link)
class float() ord() py3 only (link)
format() pow()
Built-in Exceptions

StandardError Exception BaseException

FloatingPointError
BufferError
OverflowError KeyboardInterrupt SystemExit GeneratorExit
ArithmeticError
ZeroDivisionError StopIteration

AssertionError Warning

DeprecationWarning
AttributeError
IOError
PendingDeprecationWarning
EnvironmentError
OSError
RuntimeWarning
EOFError WindowsError (Windows)
SyntaxWarning
VMSError (VMS)
ImportError
UserWarning
LookupError IndexError
FutureWarning
KeyError
MemoryError ImportWarning
NameError UnboundLocalError
UnicodeWarning
ReferenceError
BytesWarning
RuntimeError NotImplementedError

SyntaxError IndentationError TabError

UnicodeDecodeError
SystemError
UnicodeEncodeError
TypeError

ValueError UnicodeError UnicodeTranslateError

Standard Library Services

● Text Processing Services (String Services)

● Binary Data Services ● Software Packaging and Distribution
● Data Types ● Python Runtime Services
● Numeric and Mathematical Modules ● Custom Python Interpreters
● Functional Programming Modules ● Restricted Execution
● File and Directory Access ● Importing Modules
● Data Persistence ● Python Language Services
● Data Compression and Archiving ● Miscellaneous Services
● File Formats ● MS Windows Specific Services
● Cryptographic Services ● Unix Specific Services
● Generic Operating System Services ● Mac OS X specific services
● Concurrent Execution (Optional Operating System Services)
● Interprocess Communication and Networking
● Internet Data Handling
● Structured Markup Processing Tools
● Internet Protocols and Support
● Multimedia Services
● Internationalization
● Program Frameworks
● Graphical User Interfaces with Tk https://docs.python.org/2/library/index.html
● Development Tools https://docs.python.org/3/library/index.html
● Debugging and Profiling
Useful 3rd Party
Python Packages
Useful 3rd Party Python Packages

● Binary executable distribution Logging Faster Python

○ cx-Freeze ○ pex ● logging (standard library) ● Cython
○ py2exe ● logbook ● PyPy
○ pyinstaller Unittest
● Command line argument parsing ● unittest
○ argparse (py2+ standard library) ● nose
○ click ● pytest
○ docopt Code Coverage
● Web microframeworks ● coverage (pytest-cov)
○ Bottle Coding Style/Lint
○ Flask ● pep8, pycodestyle, pytest-pep8
○ Django ● flake8, pyflakes, pytest-flake8
○ CherryPy ● pylint, pytest-pylint
○ web2py
○ web.py
● xML Manipulation
○ lxml
○ xmltodict
● Template Engines
○ Template String
○ Jinja2
○ Mako
○ Genshi
Useful 3rd Party Python Packages (continued…)

● Configuration ● Image Processing

○ ConfigParser ○ pillow (fork from PIL)
○ PyYAML ● GUI
○ lxml ○ tkinter
● Relational Database ○ PyGtk
○ sqllite3 ○ PyQt / PySide
○ SQLAlchemy ○ wxPython
○ pg8000 → PostgreSQL ○ curses
○ psycopg2 → PostgreSQL ○ kivy
○ PyMySQL → MySQL ● Testing Frameworks
○ peewee ○ doctest
● NoSQL Database ○ unittest
○ PyMongo ■ nose
○ MongoEngine ○ Pytest
○ tinydb ● Debugging
● Build Tool ○ pdb
○ buildout ○ ipdb
○ buildbot ○ winpdb
● Documentation (https://readthedocs.org/) ○ pudb
○ Sphinx ○ Bugjar
○ reStructuredText ● Foreign Function Interface
○ MkDocs ○ ctypes
○ cffi
Useful 3rd Party Python Packages (continued…)

● Science and Data Analysis

○ SciPy
○ NumPy
○ NetworkX
○ Pandas
○ SymPy
○ PyCUDA
● Data Visualization
○ matplotlib (http://matplotlib.org/gallery.html)
○ bokeh (http://bokeh.pydata.org/en/latest/docs/gallery.html)
○ plotly
○ pyqtgraph
○ pyopengl
● Computer Vision
○ OpenCV
○ SimpleCV
● Excel addins
○ pyxll
○ xlwings
Application/Package
Organization
Python Modules and Packages

Python Modules
● File as python module
○ m.py → (import m) → __pycache__/m.pyc
● Directory as python module:
○ m/__init__.py (import m) → m/__pycache__/__init__.pyc

Python Packages are collection of:

● Python modules
● Programs/scripts
● Documentation
● Test
● …
Simple Single-file Zip Application and Sample Directory Structure

$ mkdir -p app/app $ tree -aF --charset ascii app

$ touch app/app/__init__.py app/app/cli.py app/__main__.py app/requirements.txt
$ cd app app
$ /opt/tools/wh/dtd/RHE64-6/python/3.6.3/bin/python3.6 \ |-- app/
-m pip install --upgrade --prefix . --requirement requirements.txt | |-- cli.py
$ cd .. | `-- __init__.py
|-- lib/
# Python2 | `-- python3.6/
$ zip -r -0 app.zip app/ | `-- site-packages/
$ echo '#!/opt/tools/wh/dtd/RHE64-6/python/3.6.3/bin/python3.6' \ | `-- click/
| cat - app.zip > app.pyz |-- __main__.py
$ rm -rf app.zip |-- __pycache__/
$ chmod +x app.pyz `-- requirements.txt

# Python3 (python -m zipapp) $ python -m app

$ cd .. $ python app/__main__.py
$ python -m zipapp \
--python /opt/tools/wh/dtd/RHE64-6/python/3.6.3/bin/python3.6 \ $ ./build/app.pyz
--output app.pyz app $ python build/app.pyz
$ chmod +x app.pyz $ PYTHONPATH=build/app.pyz python -m app
$ python -m app
$ unzip -v app/build/app.pyz
Archive: app2.pyz
warning [app2.pyz]: 55 extra bytes at beginning or within zipfile
(attempting to process anyway)
Length Method Size Cmpr Date Time CRC-32 Name
-------- ------ ------- ---- ---------- ----- -------- ----
1153 Stored 1153 0% 09-19-2017 13:08 22fb4b35 Makefile
530 Stored 530 0% 09-19-2017 11:39 73cd032c __main__.py
11 Stored 11 0% 09-19-2017 11:33 386bc06a requirements.txt
0 Stored 0 0% 09-19-2017 12:54 00000000 app/
175 Stored 175 0% 09-18-2017 15:34 408eed3c app/__init__.py
397 Stored 397 0% 09-19-2017 12:52 6bafe4cf app/cli.py
0 Stored 0 0% 09-19-2017 12:54 00000000 lib/python3.6/site-packages/click/
. ...... . .. .......
-------- ------- --- -------
412682 412682 0% 53 files
Simple Single-file Zip Application Example

$ cat app_zip/__main__.py app_zip

# -*- coding: utf-8 -*- |-- app/
# Standard Libraries
| |-- cli.py
import os | `-- __init__.py
import sys |-- lib/
from pprint import pprint
| `-- python3.6/
# Requirements path | `-- site-packages/
sys.path.append(os.path.join(os.path.dirname(__file__), 'lib', | `-- click/
f'python{sys.version_info.major}.{sys.version_info.minor}', |-- __main__.py
'site-packages'))
pprint(sys.path) `-- requirements.txt
$ cat app_zip/app/__init__.py

print(f'{__name__}, {__package__}')
"""App package"""
#!/bin/sh
if __package__ is None or __package__ == '': __author__ = 'Amal Khailtash'
from app import cli # noqa __email__ = '[email protected]' PYTHON=/opt/tools/wh/dtd/RHE64-6/python/3.6.3/bin/python
else: __version__ = '0.1.0' 3.6
from .app import cli # noqa
$ cat app_zip/app/cli.py
sys.exit(cli.main()) ${PYTHON} -m pip \
"""Command line interface.""" install --prefix ./app_zip --upgrade --requirement
app_zip/requirements.txt
# Standard libraries
import pkg_resources
${PYTHON} \
# Thirdparty libraries -m zipapp \
import click --python ${PYTHON} \
--output app_zip.pyz \
# Other libraries app_zip
from . import __author__, __version__ # noqa
# --main app:main \
def main():
print(f'{__name__}') chmod +x app_zip.pyz
print(f'Author: {__author__}, Version: {__version__}')

print(pkg_resources.resource_string(
'__main__', 'requirements.txt').decode("utf-8"))
Application/Package Directory Organization

$ tree -aF --charset ascii app

Packaging/distribution using (setup.py) distutils:
app/
├── app/
● Python packaging guide │ ├── cli.py
● The Hitchhiker's Guide to Python Packaging │ ├── __init__.py
│ └── __main__.py
● Packaging ├── AUTHORS.rst
├── build/
├── docs/
Basic project can be setup using cookiecutter or manually: │ ├── index.rst
│ ├── conf.py
$ mkdir -p app/app app/docs app/tests │ ├── Makefile
$ touch app/app/__init__.py app/tests/__init__.py │ ├── make.bat
$ cd app/docs │ ├── _build/
$ sphinx-quickstart │ ├── _static/
│ └── _templates/
Python package version management (__version__): ├── CHANGELOG.rst
├── LICENSE.rst
├── __main__.py
● versioneer ├── Makefile
├── README.rst
● bumpversion ├── requirements/
├── requirements.txt
└── tests/
├── __init__.py
└── test__???.py
Virtualenv and Managing Application/Package Requirements

$ tree -aF --charset ascii .venv

Useful aliases:
.venv
# ~/.bash_aliases |-- bin
# Create virtual environment and install pip | |-- activate
function ce () { | |-- pip
/opt/tools/wh/dtd/RHE64-6/python/3.6.3/bin/python3.6 -m venv --without-pip .venv | |-- python -> python3.6
. .venv/bin/activate | |-- python3 -> python3.6
wget -O - https://bootstrap.pypa.io/get-pip.py | python | |-- python3.6 -> /……/python/3.6.2/bin/python3.6
} |-- lib
| `-- python3.6
# Activate venv | `-- site-packages
alias ae='de; . .venv/bin/activate' |-- lib64 -> lib
`-- pyvenv.cfg
# De-activate venv
alias de='deactivate &> /dev/null'

$ cat requirements.txt
--requirements requirements/prod.txt

$ cat requirements/prod.txt
# Requirements file for production

--requirement index.txt

click==6.7
click_completion==0.2.1
click_help_colors==0.4
coloredlogs==7.3
Using virtualenv, distutils/setuptools and setup.py

$ cat setup.py $ tree -aF --charset ascii app

# -*- coding: utf-8 -*- app
|-- app/
"""Project setup""" | |-- cli.py
# from distutils.core import setup, find_packages
| `-- __init__.py
from setuptools import find_packages, setup |-- build/
|-- dist/
PARAMS = dict( | |-- app-1.0.0-py2.py3-none-any.whl
name='app', | `-- app-1.0.0.tar.gz
version='1.0.0', |-- requirements.txt
description='Description for application', |-- setup.cfg
long_description=( |-- setup.py
'Even longer description for application that does this and that.'),
author='Amal Khailtash',
`-- .venv/
author_email='[email protected]', |-- bin/
license='Ciena Proprietary', |-- include/
url='https://www.???.com', |-- lib/
keywords=['app', 'key1', 'key2'], `-- lib64 -> lib/
packages=find_packages(exclude=['docs', 'tests*']),
install_requires=['click==6.7'],
entry_points={ $ cat setup.cfg
'console_scripts': [ [wheel]
'app = app.cli:main', universal = 1
]
}, # Setup virtualenv
)
$ /opt/tools/wh/dtd/RHE64-6/python/3.6.3/bin/python3.6 -m venv \
if __name__ == '__main__': --without-pip .venv
setup(**PARAMS)
$ source .venv/bin/activate
$ wget -O - https://bootstrap.pypa.io/get-pip.py | python
$ cat app.egg-info/requires.txt
$ pip install requirements.txt
click==6.7

$ pip install --editable . # or python setup.py install

# Build and upload to PyPI

$ python setup sdist bdist_wheel upload
Parsing Command-line
Arguments
Parsing Command-Line Arguments - argparse and argcomplete

#!/usr/bin/env python $ python hello.py --count=3

# -*- coding: utf-8 -*-
Namespace(count=3, name=None)
# PYTHON_ARGCOMPLETE_OK
Your name: John
"""Simple program that greets NAME for a total of COUNT times.""" Hello John!

from __future__ import print_function

Hello John!
Hello John!
import argparse
import argcomplete
$ python hello.py --help
if __name__ == '__main__':
parser = argparse.ArgumentParser(description=__doc__) usage: hello.py [-h] [-v] [--count COUNT] [--name NAME]

parser.add_argument(
'-v', '--version', action='version', version='%(prog)s ' + __version__, Simple program that greets NAME for a total of COUNT times.
help='output version information and exit'
)
parser.add_argument('--count', type=int, help='number of greetings') optional arguments:
parser.add_argument('--name', help='the person to greet')
-h, --help show this help message and exit
argcomplete.autocomplete(parser) -v, --version output version information and exit

args = parser.parse_args()
--count COUNT number of greetings
# Use args as dictionary instead of namespace --name NAME the person to greet
# args = vars(parser.parse_args())
print(args)

count = 1 if args.count is None else int(args.count)

if args.name is None:
name = raw_input('Your name: ')
else:
name = args.name

for x in range(count):
print('Hello %s!' % name)
Parsing Command-Line Arguments - docopt

#!/usr/bin/env python $ python hello.py --count=3

# -*- coding: utf-8 -*-
{'--count': '3',
"""Usage: hello.py [--count=N] [--name=TEXT] '--name': None}
Your name: John
Simple program that greets NAME for a total of COUNT times. Hello John!
Hello John!
Arguments:
Hello John!
PATH out directory
FILE input file
$ python hello.py --help
Options: Usage: hello.py [--count=N] [--name=TEXT]
--count INTEGER number of greetings
--name TEXT the person to greet Simple program that greets NAME for a total of COUNT times.

-h, --help display this help and exit

Arguments:
-v, --version output version information and exit
NONE
"""
from __future__ import print_function
Options:
from docopt import docopt
--count INTEGER number of greetings.
if __name__ == '__main__': --name TEXT the person to greet.
args = docopt(__doc__, version='0.1')
print(args) -h, --help display this help and exit
-v, --version output version information and exit
count = 1 if args['--count'] is None else int(args['--count'])

if args['--name'] is None:
name = raw_input('Your name: ')
else:
name = args['--name']

for x in range(count):
print('Hello %s!' % name)
Parsing Command-Line Arguments - click

#!/usr/bin/env python $ python hello.py --count=3

# -*- coding: utf-8 -*-
Your name: John
import click Hello John!
Hello John!
@click.command() Hello John!
@click.option('--count', default=1, help='number of greetings.')
@click.option('--name', prompt='Your name', help='the person to greet.') $ python hello.py --help
def hello(count, name): Usage: hello.py [OPTIONS]
"""Simple program that greets NAME for a total of COUNT times."""
for x in range(count): Simple program that greets NAME for a total of COUNT times.
click.echo('Hello %s!' % name)
Options:
if __name__ == '__main__': --count INTEGER number of greetings.
hello() --name TEXT the person to greet.
--help Show this message and exit
Python Logging
Python Logging

import logging logging.basicConfig(level=logging.DEBUG)

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__) INFO:__main__:Start reading database
DEBUG:__main__:Records: {'john': 55, 'tom': 66}
logger.info('Some information...') INFO:__main__:Updating records ...
INFO:__main__:Finish updating records
# read database here
records = {'john': 55, 'tom': 66} handler = logging.FileHandler('hello.log')
logger.debug('Records: %s', records)
logger.info('More information here...') formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s -
%(message)s')
# update records here handler.setFormatter(formatter)
logger.info('Done.')
# add the handlers to the logger

INFO:__main__:Some information... logger.addHandler(handler)

INFO:__main__:More information here...
INFO:__main__:Done.
coloredlogs (https://coloredlogs.readthedocs.io/en/latest/)

import coloredlogs, logging

# Create a logger object.

logger = logging.getLogger('your-module')

# By default the install() function installs a handler on the root logger,

# this means that log messages from your code and log messages from the
# libraries that you use will all show up on the terminal.
coloredlogs.install(level='DEBUG')

# If you don't want to see log messages from libraries, you can pass a
# specific logger object to the install() function. In this case only log
# messages originating from that logger will show up on the terminal.
coloredlogs.install(level='DEBUG', logger=logger)

# Some examples.
logger.debug("this is a debugging message")
logger.info("this is an informational message")
logger.warn("this is a warning message")
logger.error("this is an error message")
logger.critical("this is a critical message")
Code Style, Lint and Static Type
Checking
pep8, pyflakes, flake8, Pylint

pep8 / pycodestyle: checks code style (--ignore=...)

$ conda/pip install pep8

$ pep8 [--statistics] [--show-source] [file name or directory name]
$ python -m pep8 [--statistics] [--show-source] [file name or directory name]

pyflakes: check syntax

$ conda/pip install pyflakes

$ pyflakes [file name or directory name]

flake8: pep8 + pyflakes + cyclomatic complexity (ignore → # noqa)

$ conda/pip install flake8

$ flake8 [file name or directory name]
$ pip install hacking # flake8 plugin with more rules from hacking (Google) style guide

Pylint: check code by importing/running (pylint: disable=....)

$ conda/pip install flake8

$ pylint [--jobs N] [module or package]
Static Type Checking

New since version 3.5 Python 2.x:

● Static types are annotated by using typing module.
def is_palindrome(s):
○ For Python 2.x it was provided through docstrings.
# type: (str) -> bool
● mypy (now a developer @Dropbox) return s == s[::-1]
○ Mypy is an optional static type checker for Python. You can
add type hints (PEP 484) to your Python programs, and use Python 3.5+
mypy to type check them statically. Find bugs in your
programs without even running them! from typing import List, Iterator
● Gradually add type hints Vector = List[float]
○ The more is added, the more checks mypy can do.
def scale(scalar: float, vector: Vector) -> Vector:
return [scalar * num for num in vector]

def is_palindrome(s : str) -> bool:

return s == s[::-1]

def fib(n: int) -> Iterator[int]:

a, b = 0, 1
while a < n:
yield a
a, b = b, a + b
Python Testing and
Test-Driven Development (TDD)
Python Testing Tools

Available unit testing packages/frameworks

● doctest - copy and paste output from shell session, tests serve as documentation.
● unittest - part of standard library similar to xUnit/jUnit framework
● pytest - automatic collection of tests;
- simple asserts
- strong support for test fixture/state management
- strong debugging support
- many plugins (pytest-cov, pytest-profiling, pytest-xdist, pytest-pep8, …)
● zope.testing - post-mortem debugging of test failures + profiling and coverage reporting

Extensions
● nose (unittest) - alternate test discovery and running process for unittest
- similar to py.test
- many plugings
Python Testing Tools - doctest

Trying:
square(2)
# doctest_example.py Expecting:
def square(x): 4
""" Return x squared. ok
>>> square(2) Trying:
4 square(3)
Expecting:
>>> square(3)
8
9 **************
""" File "square.py", line 8, in square.square
return x ** 2 Failed example:
square(3)
Expected:
8
if __name__ == '__main__': Got:
9
import doctest 1 items had no tests:
doctest.testmod(verbose=True) square
**************
1 items had failures:
1 of 2 in square.square
2 tests in 2 items.
$ python3 -m doctest -v doctest_example.py 1 passed and 1 failed.
***Test Failed*** 1 failures.
Python Testing Tools - unittest

# unittest_unittest_example.py assertFalse
import unittest assertTrue
assertEqual
import doctest_example assertNotEqual
assertAmostEqual
class UnitTestExample(unittest.TestCase): assertAlmostNotEqual
assertCountEqual
def setUp(self): assertDictEqual
print('setUp') assertListEqual
assertMultilineEqual
def tearDown(self): assertSequenceEqual
print('tearDown') assertSetEqual
assertTupleEqual
def test(self): assertIn
x = 2 assertRaises
self.assertEqual(square(x), 4)

$ python3 -m unittest -v unittest_example.py

Python Testing Tools - pytest

# test_square.py $ py.test --verbose --doctest-modules [--flake8 --pylint --cov]

def func(x): ==== test session starts =========================
return x + 1 collected 3 items

test_sample.py::test_answer FAILED
def test_answer(): test_square.py::test_square PASSED
assert func(3) == 5 test_square.py::test_exception PASSED

=== FAILURES ============

# test_square.py ____ [doctest] square.square ___
import pytest 005 Return x squared.
006 >>> square(2)
007 4
from square import square 008 >>> square(3)
Expected:
8
@pytest.fixture Got:
def setup(): 9
"""Do some test setup""" square.py:8: DocTestFailure
___ test_answer _________
def test_square(): def test_answer():
assert square(4) == 16 > assert func(3) == 5
E assert 4 == 5
def test_exception(): E + where 4 = func(3)
with pytest.raises(ValueError) as exception_info:
raise ValueError('message') test_sample.py:7: AssertionError
==== 1 failed, 2 passed in 0.10 seconds ====
assert str(exception_info.value) == 'message'
Python Documentation and
Documentation-Driven
Development (DDD)
Documentation Tools

● reStructuredText → RST (lightweight markup + semantics : e.g.: html + css)

○ http://rst.ninjs.org/
○ http://rst.ninjs.org/?n=8d2b60ae7defc38f670fd44fc9f0e8a5&theme=basic
○ .. directive-name::
○ Custom directives $ mkdir docs
○ Inline markups (:pep: `8`) $ cd docs
$ sphinx-quickstart
● Docutils
● Sphinx project/
○ Lots of extensions docs/
conf.py
○ Cross-referencing + toc indexing Makefile
○ Code markups → :rfc: `1984`, :pep: `8` index.rst
tutorials.rst
○ Syntax highlighting → .. source-code:: ruby ...
○ Autodocs (docstrings) → .. autoclass:: .. automodule::
○ html, pdf, epub, ... $ make html
$ make singlehtml
● Read the Docs (continuous integration for documentation) $ make man
○ Hosted documentation $ make doctest
○ Automatic triggers from github
○ Markdown support
○ Automatic translation + localization http://sphinx-tutorial.readthedocs.io/
○ Search + indexing
Python Code Profiling and
Performance Improvement
Measuring Code Resource Usage

Resource: # --- Manual ---------------------------------------------------

import time
● CPU
● RAM t_beg = time.time() # wall-clock or time.clock() for CPU time

● I/O (Disk, Network) # Do some work here

● … t_end = time.time() # wall-clock or time.clock() for CPU time

print(f'Execution took: {t_end - t_beg:.4} sec')

CPU resource usage measurements:
● Manual
# --- timeit ---------------------------------------------------
● timit module import timeit
● profile/cProfile module
# Default number = 1,000,000
● Third Party profilers timeit.timeit(stmt='x ** 5', setup='x = 0', number=100)

min(timeit.repeat(stmt='x ** 5', setup='x = 0', number=100, repeat=10))

IPython
● %time
● %timeit
Python Code Profiling

Use profile or faster cProfile module: #

import cProfile
$ python -m profile script.py
$ python -m cProfile script.py pr = cProfile.Profile()
r = pr.run(...)
main r.print_stats()
[] r.dump_stats('results.prof')
2.7.11 |Continuum Analytics, Inc.| (default, Dec 6 2015, 18:08:32)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)]
3 function calls in 0.000 seconds

Ordered by: standard name

ncalls tottime percall cumtime percall filename:lineno(function)

1 0.000 0.000 0.000 0.000 prog.py:4(<module>)
1 0.000 0.000 0.000 0.000 prog.py:8(main)
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}

Visual Python profiler

$ pip install -U vprof
$ vprof <modes> <test_program> # c - flame chart, m - memory graph, h - code heatmap.
Alternative Python Implementations

Alternative implementations:

● IronPython (Python running on .NET)

● Jython (Python running on the Java Virtual Machine)
● PyPy (A fast python implementation with a JIT compiler) - up to 8x faster
● Stackless Python (Branch of CPython supporting microthreads)
● MicroPython (Python running on micro controllers)

Other options:

● Python C/C++ Extension

● Instant (Python module that allows for instant inlining of C and C++ code in Python. It is a small Python module built on
top of SWIG and Distutils.)
● Numba (Numba gives you the power to speed up your applications with high performance functions written directly in
Python. With a few annotations, array-oriented and math-heavy Python code can be just-in-time compiled to native
machine instructions, similar in performance to C, C++ and Fortran, without having to switch languages or Python
interpreters).
● Cython (Cython is an optimising static compiler for both the Python programming language and the extended Cython
programming language. It makes writing C extensions for Python as easy as Python itself.) -
http://www.behnel.de/cython200910/talk.html
○ Cython can also be used to wrap external C/C++ libraries (like ctypes, swig, CFFI, pybind11)
Manually Write Extension in C/C++

https://docs.python.org/3/extending/extending.html
Instant

Creates a swig interface (.i) and wrapper (.py), compiles it into .so

# ~/.cache/instant/python3.6/cache/
from instant import inline

add_func = inline("double add(double a, double b){ return a+b; }")

print("The sum of 3 and 4.5 is ", add_func(3, 4.5))

Numba

Used for array/math heavy and JIT.

1. from numba import jit
2.
3. @jit
4. def fib_seq_numba(n):
5. if n < 2:
6. return n
7. a, b = 1, 0
8. for i in range(n - 1):
9. a, b = a + b, a
10. return a
Cython

Easily build extensions. 1. from distutils.core import setup

2. from distutils.extension import Extension
3.
$ python setup.py build_ext --inplace
4. from Cython.Build import cythonize
5.
6.
7. extensions = [
8. Extension('fib_seq_cython_type', ['fib_seq_cython_type.pyx'],
9. # define_macros = [('', '')],
10. # include_dirs = [],
11. library_dirs = ['/usr/lib64'],
12. # libraries = [],
13. ),
14. ]
15.
16. setup(
17. name = 'Recursive Fibonacci',
18. ext_modules = cythonize(
19. extensions
20. ),
21. )
Python GUI Libraries
Python GUI Libraries

Pygubu (tkinter) Glade-2 designer (PyGtk)

● tkinter (built into Python distribution)
● PyGtk
● PyQt / PySide
● wxPython

wxform builder
Qt designer
TKinter Example

from tkinter import *

from tkinter import ttk

def calculate(*args):
try:
value = float(feet.get())
meters.set((0.3048 * value * 10000.0 + 0.5)/10000.0)
except ValueError:
pass

root = Tk()
root.title("Feet to Meters")

mainframe = ttk.Frame(root, padding="3 3 12 12")

mainframe.grid(column=0, row=0, sticky=(N, W, E, S))
mainframe.columnconfigure(0, weight=1)
mainframe.rowconfigure(0, weight=1)

feet = StringVar()
meters = StringVar()

feet_entry = ttk.Entry(mainframe, width=7, textvariable=feet)

feet_entry.grid(column=2, row=1, sticky=(W, E))

ttk.Label(mainframe, textvariable=meters).grid(column=2, row=2, sticky=(W, E))

ttk.Button(mainframe, text="Calculate", command=calculate).grid(column=3, row=3, sticky=W)

ttk.Label(mainframe, text="feet").grid(column=3, row=1, sticky=W)

ttk.Label(mainframe, text="is equivalent to").grid(column=1, row=2, sticky=E)
ttk.Label(mainframe, text="meters").grid(column=3, row=2, sticky=W)

for child in mainframe.winfo_children(): child.grid_configure(padx=5, pady=5)

feet_entry.focus()
root.bind('<Return>', calculate)

root.mainloop()
PyGTk Example (with glade)

import os
import sys

import pygtk
pygtk.require("2.0")
import gtk
import gtk.glade
import gobject

class MainWindow(object):

def __init__(self):
self.wTree = gtk.glade.XML('feet_to_meters.glade', None, None)
self.wTree.signal_autoconnect(self)

# Make each widget as an attribute of your object

for w in self.wTree.get_widget_prefix(''):
name = w.get_name()
assert not hasattr(self, name)
setattr(self, name, w)

self.mainWindow.show()

def on_mainWindow_delete_event(self, widget, event, data=None):

return False

def on_mainWindow_destroy(self, widget):

gtk.main_quit()

def on__btnCalculate__clicked(self, widget):

value = self.entFeet.get_text()
self.lblMeters.set_text(str((0.3048 * float(value) * 10000.0 + 0.5)/10000.0))

if __name__ == '__main__':
MainWindow()
gtk.main()
PyQt4 Example

#!/bin/env python

from PyQt4 import QtCore, QtGui, uic

class MainWindow(QtGui.QDialog):

def __init__(self, *args):

super(MainWindow, self).__init__(*args)
self.ui = uic.loadUi("mainwindow.ui", self)

@QtCore.pyqtSlot()
def on_btnCalculate_clicked(self):
value = self.entFeet.text()
self.lblMeters.setText(str((0.3048 * float(value) * 10000.0 + 0.5)/10000.0))

if __name__ == '__main__':

import sys
app = QtGui.QApplication(sys.argv)
window = MainWindow()
window.show()
sys.exit(app.exec_())