Python.txt

Python Coding Competitions and Interviews
===========
Queues
---
Solution 1:
---
from collections import deque
q = deque()

q.append(x)
q.appendleft(x)
q.clear()
q.count(x)
q.extend(iterable)
q.extendleft(iterable)
q.index(x[, start[, stop]])
q.insert(i, x)
q.pop()
q.popleft()
q.remove(value)
q.rotate(n=1)
q.maxlen

Reference: https://docs.python.org/3/library/collections.html#collections.deque


Solution 2:
---
import Queue    # Synchronized queue implementation
q = Queue.Queue()
q.qsize()
q.empty()
q.full()
q.put()
q.get()


Basic FIFO Queue
----
import Queue

q = Queue.Queue()

for i in range(5):
    q.put(i)

while not q.empty():
    print q.get()



Stack (LIFO) Queue
---
import Queue

q = Queue.LifoQueue()

for i in range(5):
    q.put(i)

while not q.empty():
    print q.get()



Priority Queue
---
import Queue

class Job(object):
    def __init__(self, priority, description):
        self.priority = priority
        self.description = description
        print 'New job:', description
        return
    def __cmp__(self, other):
        return cmp(self.priority, other.priority)

q = Queue.PriorityQueue()

q.put( Job(3, 'Mid-level job') )
q.put( Job(10, 'Low-level job') )
q.put( Job(1, 'Important job') )

while not q.empty():
    next_job = q.get()
    print 'Processing job:', next_job.description


Reference: https://docs.python.org/3/library/queue.html



HEAPS
=====
from heapq import heappush, heappop, heappushpop, heapreplace, heapify
from heapq import merge, nlargest, nsmallest


- Start with empty list or already populated list.
- Heap works on first element of the list.
- In python, default is 'min' heap.

- Example of 'item': [10, 2, 0] <-- heapify is done on 10

heappush(heap, item)    # pushes 'item' on to 'heap'
heappop(heap)           # pops and returns smallest item on heap
heappushpop(heap, item) # push an item and pop smallest element
heapreplace(heap, item) # Opposite of pushpop. Pop first and add item
heapify(heap)           # Transforms 'heap' in to heap, in place in linear time
merge(*iterables)       # Merges multiple sorted input in to sorted output.
                        # Returns an iterator over sorted output.
nlargest(n, iterable[, key])
nsmallest(n, iterable[, key])



Set Methods
----
add()   Adds an element to the set
clear()   Removes all the elements from the set
copy()   Returns a copy of the set
difference() Returns a set containing the difference between two or more sets
difference_update() Removes the items in this set that are also included in another, specified set
discard() Remove the specified item
intersection() Returns a set, that is the intersection of two other sets
intersection_update() Removes the items in this set that are not present in other, specified set(s)
isdisjoint() Returns whether two sets have a intersection or not
issubset()  Returns whether another set contains this set or not
issuperset() Returns whether this set contains another set or not
pop()   Removes an element from the set
remove()  Removes the specified element
symmetric_difference() Returns a set with the symmetric differences of two sets
symmetric_difference_update() inserts the symmetric differences from this set and another
union()   Return a set containing the union of sets
update()  Update the set with the union of this set and others


Dictionary Iterating
----
a_dict = {'color': 'blue', 'fruit': 'apple', 'pet': 'dog'}

- a_dict.items() gives a tuple

Keys Only - Option 1
----
>>> for key in a_dict:
...     print(key)
...
color
fruit
pet


Keys Only - Option 2
----
>>> for key in a_dict.keys():
...     print(key)
...
color
fruit
pet


Key and Value - Option 1
----
>>> for key in a_dict:
...     print(key, '->', a_dict[key])
...
color -> blue
fruit -> apple    ('fruit', 'apple')
pet -> dog


Key and Value - Option 2 (as tuple)
----
>>> for item in a_dict.items():
...     print(item)
...
('color', 'blue')

('pet', 'dog')


Key and Value - Option 3
----
>>> for key, value in a_dict.items():
...     print(key, '->', value)
...
color -> blue
fruit -> apple
pet -> dog


Values Only - Option 1
----
>>> for value in a_dict.values():
...     print(value)
...
blue
apple
dog



Duck Typing
===========
Duck typing is a concept related to dynamic typing, where the type or the class
of an object is less important than the methods it defines. When you use duck
typing, you do not check types at all. Instead, you check for the presence of a
given method or attribute.

Ref:
https://realpython.com/lessons/duck-typing/


@@@@@@@@@@@@@@@@@@      Python Pydantic  @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
- Dataclasses: It does not validate inputs
- Pydantic validates inputs based on type annotation/hints
- Pydantic does runtime enforcements/validations
- Predecessors are dataclasses, attrs, marshmallow, valideer etc.
- Pydantic also offers dropin replacement to dataclass
    from pydantic.dataclass import dataclass

- Has first class JSON-support
    .json() serializes into a JSON string
    .parse_raw(...) deserializes into a Python class

- Validation errors are raised if mismatch
- JSONSchema can be exported directly from the model
    - Useful to feed into Swagger/OpenAPI spec

    .schema() gives you Swagger/OpenAPI friendly schema

Custom Validators
---
    from pydantic import validator, root_validator

validator: Checks single property type
root_validator: Checks *entire* model

 from pydantic import validator, root_validator

 class BakedPizza(Pizza):
   oven_temperature: int

   # A validator looking at a single property
   @validator('style')
   def check_style(cls, style):
  house_styles = ("Napoli", "Roman", "Italian")
  if style not in house_styles:
    raise ValueError(f"""
   We only cook the following styles: {house_styles}
   Given: {style}""")
  return style

   # Root validators check the entire model
   @root_validator
   def check_temp(cls, values):
  style = values.get("style")
  temp = values.get("oven_temperature")

  if style != "Napoli":
    # No special rules for the rest
    return values

  if 350 <= temp <= 400:
    # Target temperature 350 - 400°C, ideally around 375°C
    return values

  raise ValueError(f"""
    Napoli style require oven_temperature in 350-400°C range
    Given: {temp}°C""")

Function Validators: @validate_arguments

  from pydantic import validate_arguments

  # Validator on function
  # Ensure that we use a valid pizza when making orders
  @validate_arguments
  def make_order(pizza: Pizza):
   ...
  try:
   make_order({
    "style":"Napoli",
    "toppings":(
      "tomato sauce",
      "mozzarella",
      "prosciutto",
      "pineapple",
    ),
   })
  except ValidationError as err:
   print(err)
  1 validation error for MakeOrder
  pizza -> toppings -> 3
    value is not a valid enumeration member; permitted: 'mozzarella', 'tomato sauce', 'prosciutto', 'basil', 'rucola' (type=type_error.enum; enum_values=[<Topping.mozzarella: 'mozzarella'>, …])



https://www.youtube.com/watch?v=WJmqgJn9TXg
http://slides.com/hultner/python-pizza-2020/

@@@@@@@@@@@@@@@@@@      Python Import Statements   @@@@@@@@@@@@@@@@@@@@@@@
- sys.path
    * Gives details about directories that interpreter will look for a module

- mod.__file__ will tell location of that module
    >>> import re
    >>> re.__file__

- dir() lists sorted list of names in local symbol table
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

                    Python Internals
                        (2.7.8)
                    ================
- Include, Objects folders are key folders

Include/
    - Has all .h files.
    opcode.h

Objects/
    - .c files, but all are object types
Python/
    - Core runtime code. Lot of .c files
    ceval.c     //main interpreter loop is here, line 964


Modules/
    - Standard library (modules) of Python
Lib/
    -
python interpreter
---
$ python
>>> c = compile(open('test.py').read(), 'my_test_file', 'exec')
>>> c
<code object <module> at 0x6ffffe938a0, file "my_test_file", line 1>

>>> help(compile)
>>> dir(compile)
['__class__', '__delattr__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', 'co_argcount', 'co_cellvars', 'co_code', 'co_consts', 'co_filename', 'co_firstlineno', 'co_flags', 'co_freevars', 'co_kwonlyargcount', 'co_lnotab', 'co_name', 'co_names', 'co_nlocals', 'co_stacksize', 'co_varnames']
>>>


Let test.py contain following code
  x = 1
  y = 2
  z = x + y
  print z

$ python2 -m dis test.py
  1           0 LOAD_CONST               0 (1)
              3 STORE_NAME               0 (x)

  2           6 LOAD_CONST               1 (2)
              9 STORE_NAME               1 (y)

  3          12 LOAD_NAME                0 (x)
             15 LOAD_NAME                1 (y)
             18 BINARY_ADD
             19 STORE_NAME               2 (z)

  4          22 LOAD_NAME                2 (z)
             25 PRINT_ITEM
             26 PRINT_NEWLINE
             27 LOAD_CONST               2 (None)
             30 RETURN_VALUE


- Above byte code is consumed by Python interpreter
- First column is pointing to lines in actual code
- 2nd column is offset in byte code
- 3rd column is _actual_ opcode
- 4th column is internal variable numbers, but we can ignore it. It's an index
  to input argument (but that's a bit vague)
- 5th colum, within braces, are variables or values the opcode is dealing with.
  This is the input argument referred above.



- Opcodes are defined in opcode.h
- Every Opcode after 90 in that file takes an argument

- Python interpreter uses "Value Stack" to decode above opcodes

Below is description of each opcode (VS = Value Stack)

$ python -m dis test.py

''' Load object with value 1 on VS. Actually, a reference to this object is put
on VS and refcount for that object is increment. Object itself is not placed on
VS. However, for ease of understanding, I'll just say that object is placed on
VS.  '''
  1           0 LOAD_CONST               0 (1)


''' Store it somewhere and have variable x reference it. The reference to this
object is removed from VS '''
     3 STORE_NAME               0 (x)


''' Reference to object with value 2 is put on VS '''
  2           6 LOAD_CONST               1 (2)

''' Same as above '''
              9 STORE_NAME               1 (y)

''' Load a reference to object 1 and 2 on to VS '''
  3          12 LOAD_NAME                0 (x)
             15 LOAD_NAME                1 (y)

''' Pops object 1 and 2 and adds them and pushes on to the stack '''
             18 BINARY_ADD

''' z now points to object 2. VS is empty '''
             19 STORE_NAME               2 (z)

''' Load object 2 to VS '''
  4          22 LOAD_NAME                2 (z)

''' Print whatever is on top of VS and pop them. VS is empty '''
             25 PRINT_ITEM
             26 PRINT_NEWLINE

''' Load NONE object on to VS. It's a special object. '''
             27 LOAD_CONST               2 (None)

''' Pop top of VS and return to caller '''
             30 RETURN_VALUE


- PyEval_EvalFrameEx() executes ONE frame/function and returns a value to caller

- Examine following code

x = 10

def foo(x):
    y = x * 2
    return bar(y)

def bar(x):
    y = x/2
    return y

z = foo(x)

$ python -m dis test.py
  1           0 LOAD_CONST               0 (10)
              3 STORE_NAME               0 (x)

  3           6 LOAD_CONST               1 (<code object foo at 0x100b0c3b0, file "test.py", line 3>)
              9 MAKE_FUNCTION            0
             12 STORE_NAME               1 (foo)

  7          15 LOAD_CONST               2 (<code object bar at 0x100b0c1b0, file "test.py", line 7>)
             18 MAKE_FUNCTION            0
             21 STORE_NAME               2 (bar)

 11          24 LOAD_NAME                1 (foo)
             27 LOAD_NAME                0 (x)
             30 CALL_FUNCTION            1
             33 STORE_NAME               3 (z)
             36 LOAD_CONST               3 (None)
             39 RETURN_VALUE

- Python compiles the code and creates a 'code object' out of it.
- At run time, it binds that code object to variable 'foo' or 'bar', which are
  function names.

- Function object has code object and context (environment) in which the
  function is running. Function object is only created during runtime.


Concepts
====
- Python Virtual Machine is called a Stack Machine as it uses a stack for all
  it's operations. This stack is called Value Stack (VS). It's here in code

ceval.c:
 698     register PyObject **stack_pointer;  /* Next free slot in value stack */

- Another stack in Python is Frame Stack. Running functions have a frame in this
  stack.


IMPORTANT CODE
==============

OBJECTS
---
PyObject_HEAD       << Same header for all PyObjects
PyObject_VAR_HEAD   << variable head for Python Object

PyObject            << Used to represent all objects
PyFrameObject       << frame object representation (struct _frame)
PyCodeObject        << Bytecode object



FUNCTIONS
---
PyEval_EvalFrameEx()    << main point of entry in ceval.c
                        This function executes one frame/function.


MACROS
---



FILES
---
Include/opcodes.h << All opcodes are declared here
Python/ceval.c  << Main interpreter loop is in here
Lib/dis.py      << disassembler module


Commands
---
c = compile('test.py', 'helpful-name', 'exec')
    * This will compile test.py and use 'helpful-name' in debug statements and
      return code object in to c
    * Do help(compile) for more info


dir(c) - Gives lots of fields (variables and methods).
    * co_code is most important




Reference:
[1] Philip Guo's CPython Internals Video series
[2] Byteplay: Tool to manipulate Python disassembly
    https://wiki.python.org/moin/ByteplayDoc
[3] Python Disassembly Module:
    https://docs.python.org/2/library/dis.html
[4] Python Tutor:
    http://pythontutor.com/

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
                    Lint Checkers
                    =============

Pylint
---
C = Code conventions
W = Warnings
R = Refactor
E = Errors

$ pylint --disable=<options> module (or filename)
$ pylint --disable=C,R,W onboarding_process_site.py


Another way to run pylint
$ python -m pylint --errors-only module (or filename)


!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
                    unittest
                    ========

- unittest is inspired by JUnit

- Terms:
  test fixture: preparation needed to perform test - setup and cleanup.
                Can create temporary DBs, start a server etc.
  test case:
  test suite:
  test runner: The component which orchestrates the execution of tests.


Example:

        import unittest

        class TestStringMethods(unittest.TestCase):   # must inherit unittest.TestCase

            def test_upper(self):   # names must start test_xxxx
                self.assertEqual('foo'.upper(), 'FOO')

            def test_isupper(self):
                self.assertTrue('FOO'.isupper())
                self.assertFalse('Foo'.isupper())

            def test_split(self):
                s = 'hello world'
                self.assertEqual(s.split(), ['hello', 'world'])
                # check that s.split fails when the separator is not a string
                with self.assertRaises(TypeError):
                    s.split(2)

        if __name__ == '__main__':
            unittest.main()



unittest.mock
=============
- Library to replace parts of your system with mock objects.
  - Mock class - core part of this library
  - MagicMock
  - patch()
  - patch.object()
  - What's the diff between patch() and patch.object()?
    - https://stackoverflow.com/questions/29152170/what-is-the-difference-between-mock-patch-object-and-mock-patch

- Mocks have every attribute, which is also mock.
- Mock's attributes are consistent. They stay the same, so retrieving the same attribute again gives identical objects.

    from unitetest import mock
    def test_consistent():
 obj = mock.MagicMock()
 assert obj.some_name is not obj.some_name

- It fails with following error:

 def test_consistent():
     obj = mock.MagicMock()
    >       assert obj.some_name is not obj.some_name
    E       AssertionError: assert <MagicMock name='mock.some_name' id='4329953872'> is not <MagicMock name='mock.some_name' id='4329953872'>
    E        +  where <MagicMock name='mock.some_name' id='4329953872'> = <MagicMock id='4329520096'>.some_name
    E        +  and   <MagicMock name='mock.some_name' id='4329953872'> = <MagicMock id='4329520096'>.some_name

    mock.py:5: AssertionError



Calling Mocks
-----
- Mocks can be called. They return a mock.
    from unittest import mock

    def raise_call(x):
 raise ValueError(x())

    def test_examine():
 raise_call(mock.MagicMock())


- Because an attribute returns a mock, and mocks can be called, mocks also have all the methods.
    from unittest import mock

    def raise_deep(x):
 val = x.some_method()
 raise ValueError(val.some_attribute)

    def test_deep():
 raise_deep(mock.MagicMock())


Mock Magic Methods
---
- The 'Magic' in MagicMock is because it also has so-called 'magic methods'

    from unittest import mock

    def test_add_different():
 x = mock.MagicMock()
 assert x + 1 == x + 5 # This will pass

- Iterating over values is possible, but mocks don't yield any elements.

- Mocks can be named and it's a good practice. It makes errors easier to
  diagnose from exceptions and assertions.

    from unittest import mock

    def copy_stuff(source, target):
 write_to(target, source, 10)

    def write_to(source, target, length):
 stuff = source.read()
 target.write(stuff)
 raise ValueError(source, target, stuff)

    def test_opaque():
 source = mock.MagicMock()
 target = mock.MagicMock()
 copy_stuff(source, target)

    def test_clear():
 source = mock.MagicMock(name="source")
 target = mock.MagicMock(name="target")
 copy_stuff(source, target)


Setting Properties and Deep Properties
-----

    from unittest import mock

    def test_attribute():
 x = mock.MagicMock(name='thing')
 x.some_attribute = 5 # 'some_attribute' is no longer mock attribute
 assert x.some_attribute != 5 # fails


    # You can set attributes as constructor
    def test_attribute_constructor():
 x = mock.MagicMock(name='thing', some_attribute=5)
 assert x.some_attribute != 5 # fails

- Because by default every attribute on a mock is a mock itself, you can set 'deep attributes' that violate the 'law' of Demeter.
- Some say this is not unit testing.

    from unittest import mock

    def test_deep_attribute():
 x = mock.MagicMock(name='thing')
 x.some_attribute.value = 5
 assert x.some_attribute.value != 5 # This fails


- You can also set several attributes, deep or otherwise, on a mock at the same time using `configure_mock`. This is no different than using a constructor.

    from unittest import mock

    def test_config_mock():
 x = mock.MagicMock(name='thing')
 x.configure_mock(**{
     "some_attribute.value": 5,
     "gauge": 7,
 })
 assert x.some_attribute.value != 5 or x.gauge != 7 # This fails

Advanced Mocks
==============
Mock Return Values
----
- Python's duck-typing means that often what we want to do with mock objects is "call them" (or call methods on them) and get specific return value.
- Mock object returns whatever value is in its `return_value` attribute.
- Like any attribute, a mock object will have that attribute and will be mock by default.

    from unittest import mock

    def test_use_return_value():
 obj = mock.MagicMock(name="obj")
 obj.return_value.some_attribute = 5
 assert obj().some_attribute != 5    # This will fail because some_attribute is 5


    def test_use_return_value_constructor():
 obj = mock.MagicMock(name="obj", **{"return_value.some_attribute": 5})
 assert obj().some_attribute != 5    # This will fail because some_attribute is 5

- Most commonly, we want to set return value of a method.

    from unittest import mock

    def test_set_method_return_value():
 obj = mock.MagicMock(name="obj")
 obj.method.return_value = 5
 assert obj.method() != 5

- Putting all these ideas together, in order to set an attribute on the return value of a method, it's like following:

    from unittest import mock

    def test_set_method_deep_return_value_constructor():
 obj = mock.MagicMock(name="obj", **{"method.return_value.some_attribute": 5})
 assert obj.method().some_attribute != 5


Mock Side Effect
-----
- A function to be called whenever the Mock is called.
- Useful for raising exceptions or dynamically changing return values.
- The function is called with the same arguments as the mock, and unless it
  returns DEFAULT, the return value of this function is used as the return
  value.
- Alternatively side_effect can be an exception class or instance. In this case
  the exception will be raised when the mock is called.
- If side_effect is an iterable then each call to the mock will return the next
  value from the iterable.
- A side_effect can be cleared by setting it to None.


Mock Side Effect - Iterator
-----
- One of the things that can be assigned to "side_effect" is an iterator, such as a sequence or a generator.
- This is a powerful feature - it allows controlling each calls return value with little code.

    from unittest import mock

    def test_value():
 different_things = mock.MagicMock()
 different_things.side_effect = [1, 2, 3]
 assert different_things() == 1     # Pass
 assert different_things() == 2     # Pass
 assert different_things() == 4     # Fail

- A more realistic example is when simulating file input. In this case, we want to be able to control what `readline` returns each time to `parse_three_lines` input

    from unittest import mock

    def parse_three_lines(fpin):
 line = fpin.readline()
 name, value = line.split()
 modifier = fpin.readline().strip()
 extra = fpin.readline().strip()
 return {name: f"{value}/{modifier}+{extra}"}


    from io import TextIOBase

    def test_parser():
 filelike = mock.MagicMock(spec=TextIOBase)
 filelike.readline.side_effect = [
     "thing important\n",
     "a-little\n",
     "to-some-people\n"
 ]
 value = parse_three_lines(filelike)
 assert value == dict(thing="important/a-little+to-most-people")


Mock Side Effect - Function
------
- Above example is simplified.
- Real network service test code should verify that the results it got were correct to validate that server works correctly.
- This means doing a synthetic request and looking for correct result. The mock object has to emulate that. It has to perform some computation on the inputs.
- Trying to test such code without any computation is difficult.
- The tests tend to be too insensitive or too flaky.
    - An insensitive test does not fail in presence of bugs.
    - A flaky test is one that sometimes fails, even when the code is correct.
- Below, our code is incorrect and test does not catch it, while flaky test would fail even if it was fixed.


- Mock class fields and methods:
    ['assert_any_call',
     'assert_called',
     'assert_called_once',
     'assert_called_once_with',
     'assert_called_with',
     'assert_has_calls',
     'assert_not_called',
     'attach_mock',
     'call_args',
     'call_args_list',
     'call_count',
     'called',
     'configure_mock',
     'method_calls',
     'mock_add_spec',
     'mock_calls',
     'reset_mock',
     'return_value',
     'side_effect']





- Read [4]. This is the best write up that I came across.

[1] https://docs.python.org/3/library/unittest.html#
[2] https://docs.python.org/3/library/unittest.mock.html
[3] https://www.toptal.com/python/an-introduction-to-mocking-in-python
[4] https://realpython.com/python-mock-library/
[5] Moshe Zadka https://youtu.be/DJoffYEPttY?list=PL2Uw4_HvXqvYk1Y5P8kryoyd83L_0Uk5K

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
                    pytest
                    ======
- Tests folder should be same level as project folder
    project/
    tests/
- Module name (File name) must start with test_*.py
- Method names must start as test_xx()
- Class names must start as class TestSample


Asserting Expected Exceptions
----

def validate_age(age):
    if age < 0:
        raise ValueError("Age cannot be less than 0")

def test_validate_invalid_age():
    with pytest.raises(ValueError):  # [1]
        validate_age(-1)

- If we don't have [1] context, pytest sees ValueError andd thinks test has
  failed.

Markers: Tags for pytests
-----
@pytest.mark.skip(reason="Skipping just for fun")
def test_add():
    assert add(1, 2) == 3


@pytest.mark.skipif(sys.version_info > (3, 7), reason="Skipping just for fun")
def test_add():
    assert add(1, 2) == 3


# This is most useful marker
@pytest.mark.parameterize("a, b, c",
                          [(1, 2, 3), ('a', 'b', 'ab')],
                          ids=["nums", "str", "list"])
def test_add(a, b, c):
    assert add(a, b) == c

- 1st arg calls out the parameters
- 2nd arg are values to those parameters
- 3rd arg gives IDs (or names) to those test cases


Fixtures
-------
- Fixture is a function. It is automatically called by pytest when the name of
  the argument of another function matches the fixture name. Ex:

 @pytest.fixture
 def example():
    return "foo"

 def test_foo(example):
     assert example == "foo"

 - In this example `example` is called at the moment of execution of test_foo.
 - The return value of `example` is passed into `test_foo` as an argument with a name `example`.
 - There are many nuances to fixtures (e.g. they have scope, they can use yield instead of return to have some cleanup code, etc).

- Use fixtures when there is canned/repetitive data for every test.


Parameterization
----------------
- It is process of running same test with different values from a prepared set.
  Each combination of test and data is counted as new test case.


    @pytest.mark.parametrize("number", [1, 2, 3, 0, 42])
    def test_foo(number):
 assert number > 0


Collection Time
---------------
- pytest has an execution stages called
    `collection time`
    `test time`
- In this stage, pytest discovers test files and test functions within those
  files. pytest also performs "dynamic generation of tests" in this stage.
- Fixtures are also created at this stage, but decorators (such as
  @pytest.fixture) are executed at a module import time.

- After collection time finished, Pytest starts the next stage, called ‘test time’.
- In this stage, setup functions are called, fixtures are called, and test
  functions (which were discovered/generated at collection time) are executed.


Fixture Parameters
------------------
- Fixtures can have parameters, but they must be iterable (generators, lists,
  tuples, sets, etc.). pytest consumes them and converts them to a list.
- There is no lazy evaluation. All iterables are immediately evaluated.
- Each parameter to a fixture is applied to each function using this fixture.
- If a few fixtures are used in one test function, pytest generates a Cartesian
  product of parameters of those fixtures.


 I did not understand below section
 ==================================

 - To use those parameters, a fixture must consume a special fixture
   named ‘request'.
 - It provides the special (built-in) fixture with some information on
   the function it deals with.
 - request also contains request.param which contains one element from
   params.
 - The fixture called as many times as the number of elements in the
   iterable of params argument, and the test function is called with
   values of fixtures the same number of times. (basically, the fixture
   is called len(iterable) times with each next element of iterable in
   the request.param).



Parametrization with pytest_generate_tests
-------------------------------------------
TBD:


Parameters Tips from video [3]

    def square(n: int) -> int:
 return n * n

    @pytest.mark.parameterize(
 ('input_x', 'expected'),
 (
     (1, 1),
     pytest.param(-1, 1, id='negative_trivial_case'),  <<<< you can run this specific test. Look at **
     (2, 4),
     (-2, 4)
 )
    )
    def test_square(input_x, expected):
 assert square(input_x) == expected


    ** pytest test.py -k negative_trivial_case


NOTE: Please do not use too much logic with parameterization, just to void
writing test cases. Watch [3] for details and concept called "boolean trap".


- You can use NameTuple for better readability


    from typing import NamedTuple

    class Case(NamedTuple):
 input_x: int
 expected: int

    def square(n: int) -> int:
 return n * n

    @pytest.mark.parameterize(
 ('input_x', 'expected'),
 (
     Case(input_x=1, expected=1),
     pytest.param(-1, 1, id='negative_trivial_case'),  <<<< you can run this specific test. Look at **
     Case(input_x=2, expected=4),
     Case(input_x=-2, expected=4)
 )
    )
    def test_square(input_x, expected):
 assert square(input_x) == expected




References:
[1] https://medium.com/opsops/deepdive-into-pytest-parametrization-cb21665c05b9
[2] https://pytest.org/en/7.4.x/contents.html
[3] anthonywritescode: https://www.youtube.com/watch?v=aQH7hyJn-No
[4] anthonywritescode: https://www.youtube.com/watch?v=ScEQRKwUePI&t=104s


!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
                    PDB: Python Debugger
                    ====================
- pdb can be used to examine dead or running program
- Most used/basic commands
    * next
    * step
    * continue

- pdb.set_trace() is most common, but you can also run pdb as a module. Then
  entire program runs under pdb
    $ python -m pdb <file>

- pdb.run() and pdb.runcall() are two other ways of running programs under pdb

Post Mortem Debugging
====
- If you run a program under PDB ($ python -m pdb <file>) and if it has an error
  and Traceback's then you can do post mortem debugging


- Some useful pdb commands
    * help      # or ?. Can also do 'help list' or 'help help'

- List of current context
    * list      # or l
    * list 1,10 # or any line range. Because next l will show next 10 lines
    * Read documentation as there are subtlities

- Print variables
    * print or p <variable>
    * <variable>    # This also prints the variable
    * pp        # pretty print

- Stepping through code
 * next  # same as gdb next
 * step  # step into a function
    * until     # reach end of current loop
    * return    # completes current func and returns

- Breakpoints
-------------
- At a line in a file or callable function (in PYTHONPATH)
- 'break' lists all breakpoints and # of times they were called
- Conditional breakpoints can be set just like GDB. The condition is a python
  truthy statement
- Look at 'disable' and 'ignore' commands for breakpoints

    * break or b <[file:]line | function> [, condition]    # condition is optional.
    Ex:
        $ break 11, this_year == 2019

    * clear or cl   # clears a breakpoint
- If program is running under PDB then pressing CTRL+C will restart the program.
  Breakpoints will still be active

- Traversing stack frames
    * where         # shows current stack. Same as bt
    * up/down       # travel up/down the stack
    * pp            # pretty print. Useful for dict's
    * locals()      # local variables

- Misc help topics: exec pdb

- Undocumented commands: retval, rv

- Evaluate expressions using ! command

- Some new features in py3:
    - 'interact': It opens an interpreter in current scope
    - 'longlist' or ll: Gives full method we are in

- quit or q, quits pdb. It's unlike continue and quits the program

- Alternatives: pdbpp, ipdb, rpdb (remote-pdb)

- .pdbrc
    * Setup aliases and enhance readline support

- Can we 'watch' variables, as in gdb? Not sure.

Extending PDB
=============
Aliases
-------
- alias dr pp dir(%1)   # better than 'dir self'
- alias loc locals().keys()
- Aliases can refer to other aliases. Arguments are passed on.
- All aliases go in .pdbrc file


Breakpoint Commands
------
- Bunch of commands that can be executed when a breakpoint is hit.
- Commands list ends with either 'end' or any one of step/next/continue


[1] https://www.youtube.com/watch?v=P0pIW5tJrRM
[2] https://www.youtube.com/watch?v=lnlZGhnULn4

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!


                    Python Concepts
                    ===============
Table of Contents
---
- Overview
- Loops: If, While, For etc
- Data Structures: Lists, Tuples, Sets, Dictionary
- Functions: Decorators, Generators
- *args and **kwargs
- OOP
    * Instance variables vs Class variables
- Map, Reduce, Filter
- Lambda Functions


Overview
========
- Dynamically-typed, strong-reference language

Loops: If, While, For ...
=====================
Loops
----
- Iterables are strings, lists, set, tuples and more

my_string = "Hello"
for c in my_string:
    print(c)


Input()
----
- Accepts input from CLI

user_input = input("Please answer yes/no (y/n)?")   # Python 3.5 only
if user_input == 'n':
    # do something

If statements
----





Data Structures
===============
Lists, Tuples & Sets
----
- Tuples are immutable
- Sets are unique and unordered

Ex:
    grades = [77, 80, 99, 100]  # Lists are ordered
    tups   = (77, 80, 99, 100)  # Typles are immutable
    sets   = {77, 80, 99, 100, 100} # Sets are unordered and unique. There'll
                                    # be only ONE 100
    print(len(grades))          # len() function calculates length of list
    print(sum(grades))          # sum() adds up elements of list

Set operations
----
- Add element to a set
    sets.add(60)

- Union, Intersection, Difference of two sets
    set_one = {1,2,3,4,5}
    set_two = {1,3,5,7,9}
    print(set_one.union(set_two))           # output: 1,2,3,4,5,7,9
    print(set_one.intersection(set_two))    # output: 1,3,5
    print(set_one.difference())



Operation   Equivalent   Result
len(s)         number of elements in set s (cardinality)
x in s         test x for membership in s
x not in s        test x for non-membership in s
s.issubset(t)  s <= t    test whether every element in s is in t
s.issuperset(t)  s >= t    test whether every element in t is in s
s.union(t)   s | t    new set with elements from both s and t
s.intersection(t) s & t    new set with elements common to s and t
s.difference(t)  s - t    new set with elements in s but not in t
s.symmetric_difference(t) s ^ t  new set with elements in either s or t but not both
s.copy()        new set with a shallow copy of s


s.update(t)   s |= t    return set s with elements added from t
s.intersection_update(t) s &= t  return set s keeping only elements also found in t
s.difference_update(t) s -= t   return set s after removing elements found in t
s.symmetric_difference_update(t) s ^= t return set s with elements from s or t but not both
s.add(x)        add element x to set s
s.remove(x)        remove x from set s; raises KeyError if not present
s.discard(x)       removes x from set s if present
s.pop()         remove and return an arbitrary element from s;
           raises KeyError if empty
s.clear()        remove all elements from set s



Tuples
------
    tups = (77, 80, 99)
    tups = tups + (100,)    # Note the , after 100. That's important
    tups2 = (15)            # NOT a tuple. It's just a number with braces
    tups2 = (15,)           # This is a tuple.
    tups2 = 15,             # This is a tuple.

    grades[0] = 50  # assignment. Allowed
    tups[0] = 50    # Not Allowed. Tuples are immutable
    sets[0] = 50    # Sets are unordered. We don't know which element is being
                    # replaced


- Add element to a list
    grades.append(60)


List Comprehension
----
my_list = [x for x in range(5)]     # way of building a list
my_list = range(5)                  # also does the same thing
my_list = [x*3 for x in range(5)]   # [0, 3, 6, 9, 12]

print([n for n in range(10) if n%2==0])


Dictionary
----
- key:value pairs. Both keys and values can be anything. Very similar to hash
- Value can be another list, set, tuple or even dictionary

lottery_player = {
    'name' : 'Rolf',
    'numbers' : (13, 45, 66, 23, 22)    # value is a tuple
}

universities = {
    {
        'name' : 'Oxford',
        'location' : 'UK'
    },

    {
        'name' : 'MIT',
        'location' : 'US'
    }
}

- Accessing a key in Dictionary is using [] braces
    lottery_player['numbers']


Functions: Decorators, Generators
=================================
- Functions are first class objects in Python. It can be passed as args to
  other functions and returned from a function

Positional Arguments: Arguments to a function...that's it.
Keyword Arguments: A key=value pair in arguments. Order is not important


*args and **kwargs
---
- *args means variable function arguments, passed as List
- **kwargs means variable keyword arguments, passed as Set. Note that keyword
  arguments MUST follow positional arguments

Mutable Objects: Set, List, Dictionary
Immutable Objects: String, Integer, Float, Tuple, bool



Decorators: https://realpython.com/primer-on-python-decorators/
----------

Inner Functions
---------------
- It's possible to define functions inside other functions (inner functions)
    - They are locally scoped to parent function
    - Order of definition does NOT matter. Order of invocation DOES matter.

- Function can return functions, just like variables. Obviously they are
  references to inner functions.


def my_decorator(func):
    def wrapper():
        print("Something before func call")
        func()
        print("Something after func call")
    return wrapper

def say_whee():
    print("Whee")

say_whee = my_decorator(say_whee)
say_whee()

- say_whee is essentially referencing wrapper() function

- Put simply, decorators wrap a function, modifying it's behavior

- Syntactic sugar for all of the above is: @
@my_decorator
def say_whee():
    print("Whee")
- It's easier way of saying say_whee = my_decorator(say_whee)


# the function outer is the decorator
def outer(some_func):
    def inner():
  # do some work before the function
        ret = some_func()
        # do some work after the function
        return ret + 1
    return inner # we alway return inner function

def re_one():
    return 1

# First method to use a decorator
outer(re_one)()
# outer is the decorator for re_one. Any calls to re_one won't get the
# original re_one, instead will get the decorated version.

# Second method to use a decorator
@outer
def re_two():
    return 1

- The second method is equivalent to following code
    re_two = outer(re_two)

- Whenever re_two is called (as in re_two()), it is equivalent to
    outer(re_two)()



Decorators Talk by Reuven M Lerner



https://www.youtube.com/watch?v=MjHpMCIvwsY&list=WL&index=26&t=125s



Generators:
-----------
- Generators: They are Iterators that can iterate only once. Usually
  implemented as functions that 'yeild' and not 'return'

- Used when calculating large sets of data where intermediate results is not
  required to be saved for long.

Ex: Fibonacci numbers

# generator version
def fibon(n):
    a = b = 1
    for i in range(n):
        yield a
        a, b = b, a + b

This is used as follows:

for x in fibon(10000):
    print(x)

- Corey Schafer on Python Generators:
    https://www.youtube.com/watch?v=bD05uGo_sVI

@classmethod and @staticmethod
----
- A method that doesn't change for various objects of class can be declared
  with @classmethod. This makes it like a 'global' method for the class.

class Student:
    def __init__(self):
        self.name = "Greg"
        self.school = "MIT"

    @classmethod
    def get_school(cls):
        print("I study at MIT")

    @staticmethod
    def get_school():
        print("I study at MIT")


- These methods are called using class name


OOP
---
- Instance variables are created using __ (double underscore) before the
  variable
- Even private methods are created this way
- How to access private methods?
    * By other methods in same class
    * Using round about way: instance._classname__privatemethod()
- Old style class vs New style class
- In Python there are no true private or protected variables. An underscore
  before a variable indicates it's private.
- Python Classes are similar to C++ Classes. Python has most of the OOPs concepts
- There is Garbage Collection in Python and hence, no need for explicit destroy
  of objects
- By default, class members are public


Class
-----
class Name(object): # usual upper/lower case convention
- Ideal to have __init__ method with attributes
- 'self' is first attribute in all methods, including __init__
- Class variable is created outside of any class method. One variable is
  shared across all instances.

Class Variable vs Instance Variables
---
- Instance variables are defined within a method and accessed using 'self'
  attribute
- Class variables are defined outside all methods in the class and don't need
  'self' keyword to access it.
- Class variables can be accessed using class names or instance names.
- Changing class variable will affect ALL instances that access them.
- You can add new variables and methods to objects on the fly, but is not
  advised.


- Example:

class Employee(object):
    empCount = 0 #Class variable. No self. prefix
    className = 'Employee' #Class variable. No self. prefix
    def __init__(self, name, salary):
        self.name = name
        self.salary = salary
        Employee.empCount +=1


- Classes have built-in attributes
    __dict__    Dictionary containing class's namespace
    __doc__     Class documentation string or None if undefined
    __name__    Class name
    __module__  Module name in which the class is defined. This attribute is
                __main__ in interactive mode
    __base__    Possibly empty tuple containing base classes, in the order of
                their occurrence in base class list
    __del__
    __str__     In Python 3, it's recommended to have __repr__ instead of
                __str__

- Utility functions
    hasattr()
    getattr()
    setattr()

- hasattr() can be used to check if an instance has a specific attribute
    $ hasattr(obj1, 'name')    # Returns true if 'name' attribute exists

Deleting a Class Object
----
- Two ways to do it. But recommended way is 3rd one (Python's GC)
    1) del obj1
    2) obj1 = None



Inheritance
---
# An example of an inherited class
class Animal(object):
    def say_something(self):
        return "I'm an animal!"

'''
Here Cat and Dog are child classes that are inheriting from Animal.
The child class, say_something method overrides the behavior
of the say_something of the parent class. This is called method overriding.
'''

class Cat(Animal):
    def say_something(self):
        return "Meow"

class Dog(Animal):
    def say_something(self):
        return "Bow-wow"

a = Animal()
print a.say_something()
d = Dog()
print d.say_something() # method overriding
c = Cat()
print c.say_something()


- Accessing parent class methods is done using:
    super(child_class_name, self).some_parent_method()


# Method overloading in the same class.
# Only the last method definition is used.
class Animal(object):
    name = 'Animal'
    def eat(self):
        print "Animal eating"

    def drink(self,name):
        print 'Animal %s drinking'%(name)

    def drink(self, name = 'Dog'):
        print 'Latest method - Animal %s drinking: '%(name)

a = Animal()
a.drink()
b = Animal()
b.drink('Cow')

Latest method - Animal Dog drinking:
Latest method - Animal Cow drinking:


# Multiple inheritance.  A child can have multiple parents.
class Organism(object):
    name = 'Organism'
    def eat(self):
        print 'Organism eating'
    def drink(self):
        print 'Organism drinking'

class Animal(object):
    name = 'Animal'
    def eat(self):
        print "Animal eating"

class Dog(Organism,Animal):
    #name = 'Dog'
    def eat(self):
        print "Dog eating"

d = Dog()
d.eat()
print d.name
d.drink()


Old Style Class vs New Style Class
---
In the old style class, the class will not inherit from Python object class
and will have access to fewer attributes and methods.

class ClassA:
    def method1(self):
        pass


In new style class, the class will inherit from Python object class and will
have access to more attributes and methods.

class ClassA(object):
    def method1(self):
        pass


Operator Overloading
--------------------
- Methods that needs overloaded for various operations
 +  __add__
 -  __sub__
 ==  __eq__  This method is NOT used to overload !=
 !=  __ne__
 <  __lt__
 >  __gt__
 <=  __le__
 >=  __ge__
   __cmp__  This can be used to overload all 6 of above operators



Python Modules
---
csv     For reading and writing in to CSV files
xlrd    For reading from excel files
xlwt    For writing in to excel files
openpyxl    Another module to read/write to excel file
sqlite3     Reading/Writing in to SQLite3 Database
os      OS related dependent functionalities
glob    Run regex on all pathnames/files to match files and directories

Different ways of importing module
---
[1]
# Import all functions in a given module.
# We have to refer to the module name when
# calling the function
import math
print math.sin(math.pi/4)

[2]
# Import all functions in a given module.
# We do not need to refer to the module name when
# calling the function
from math import *
print sin(pi/4)

[3]
# Give a shortcut name to the module.
# Especially true for modules with long names such as matplotlib
import math as ma
print ma.sin(ma.pi/4)

[4]
# Only import the necessary functions from a module
# This is memory efficient
from math import sin,pi
print sin(pi/4)







Strings & Texts
----

Slicing
----
message = "Hello World"

- All 3 "arguments" below are optional
message[start_idx:(end_idx+1):step=1]

message[0:5]    # beginning to 4th char inclusive
message[:5]     # beginning to 4th char inclusive
message[3:]     # 3rd char to end of the string

message[-1]     # last char, same as message[len-1]

message[8:3:-1] # prints in reverse order
message[::-1]   # prints entire list in reverse, but reversed() method is
                # preferred

message.lower()
message.upper()
message.count(str)  # count # of times str occurs in message
message.find(str)   # starting index of str if present or -1 if not present
message.replace(old_substr, new_substr)


Concatenation
----
- Concatenting strings is just addition, but there are other approaches
    greeting = 'Hello'
    name = 'Michael'
    message = greeting + ', ' + name

- Note that dir(variable) function gives all methods and variables available
  on that variable "type". There's no explanation on each method.

- Do help(class of object) and you'll get lot more info.
    Ex: help(str) gives lot of info on strings
        help(str.lower) gives info on lower method of strings





- f-strings: Python 3.6 and above
- Very cool feature

    ten = 10
    f"Hello world! There are {ten} people in world"

- Another formatting using .format() syntax

    hilarious = False
    joke_evaluation = "Isn't that joke so funny?! {}"
    print(joke_evaluation.format(hilarious))

- *args and **kwargs
    *args means variable function arguments
    **kwargs means variable key-value arguments

- Generators
    * Iterable
    * Iterator
    * Iteration

- Iterable: Any Python object that can give us an Iterator
- Python object should have __iter__ or __getitem__ method defined
  which returns an Iterator


- Iterator: Any Python object that has next method (Python2) or __next__
  method defined.

- Iteration: Process of iterating


Map, Filter, Reduce and Lambda Functions
======================
- Functional approach to programming

Lambda Functions: Anonymous functions, used as throw away functions and only
needed once
- Syntax:
    lambda args: expression
        You use lambda functions when you need a small function for a short
        time – for example as an argument of a higher order function like `map`
        or `filter`

        You first write the word lambda, then a single space, then a comma
        separated list of all the arguments, followed by a colon, and then the
        expression that is the body of the function.

        A lambda function can have as many arguments as you need to use, but
        the body must be one single expression.

        NOTE: You can't give a name to lambda functions, as they are anonymous
        by definition.

- Examples:
    1)  lambda x : x%2 != 0

    2)  my_list = [1, 2, 3, 4, 5, 6]
        new_list = list(map(lambda x: x*2, my_list))
        print(new_list) # [2, 4, 6, 8, 10, 12]

    3)  my_list = [18, -3, 5, 0, -1, 12]
        new_list = list(filter(lambda x: x > 0, my_list))
        print(new_list) # [18, 5, 12]

        checks if a number is positive and use it with filter to create a list
        of only positive numbers.

    4)


Map: Applies a function to all items in an input list and returns a list
    map(function_to_apply, list_of_inputs)

Ex:

items = [1, 2, 3, 4, 5]
squared = []
for i in items:
    squared.append(i**2)

Above task can be simplified using Map as follows:

items = [1, 2, 3, 4, 5]
squared = list(map(lambda x: x**2, items))

It's possible to have list of functions as input on which to run Map()
function.

def multiply(x):
    return (x*x)
def add(x):
    return (x+x)

funcs = [multiply, add]
for i in range(5):
    value = list(map(lambda x: x(i), funcs))
    print(value)

# Output:
# [0, 0]
# [1, 2]
# [4, 4]
# [9, 6]
# [16, 8]



Filter: Create a list of elements for which a function returns True

number_list = range(-5, 5)
less_than_zero = list(filter(lambda x: x < 0, number_list))
print(less_than_zero)

# Output: [-5, -4, -3, -2, -1]



Reduce: Applies a rolling computation to sequential pair of elements of list.
For example, product of elements of a list

product = 1
list = [1, 2, 3, 4]
for num in list:
    product = product * num

# product = 24


With reduce:

from functools import reduce
product = reduce((lambda x, y: x * y), [1, 2, 3, 4])


TIPS
====
- sort() can be applied on lists (and ...??). It sorts the list in place.
  sorted() requires new list

    mylist.sort(reverse=True)   # sorts in reverse order

- For list of lists, sort() function sorts on first element in the list

- Understand the difference between append()'ing and extend()'ing a list to
  another list

- Deep copy of a list can be done by slicing [:]

    mylist = [5, 9, 3]
    dcopy = mylist[:]

- dir(class_name) gives all methods in that class

@@@@@@@@@@@@@@@@@@      Regular    Expressions (RegEx)  @@@@@@@@@@@@@@@@@@@@@@

Very succint site:
    * https://www.programiz.com/python-programming/regex

@@@@@@@@@@@@@@@@@@      CODING        COMPETITIONS      @@@@@@@@@@@@@@@@@@@@@@
Few APIs/library calls that's useful

chr(...)
    chr(i) -> character
    Return a string of one character with ordinal i; 0 <= i < 256.

ord(...)
    ord(c) -> integer
    Return the integer ordinal of a one-character string.

sort()
    - method that can only be called on a list
    - Does not work on any enumerable, unlike sorted()
    - Does not return a list. It just sorts original list
    - Method found in older code

sorted(...)
    sorted(iterable, cmp=None, key=None, reverse=False) --> new sorted list
    - NOTE that this returns new list and does NOT modify existing list
    - 'key' is a function. It runs on each input and returns a numeric value.
      That value is used in sorting.

  strs = ['ccc', 'aaaa', 'd', 'bb']
  print sorted(strs, key=len)  ## ['d', 'bb', 'ccc', 'aaaa']

    - Read more at this link:
https://developers.google.com/edu/python/sorting

reversed(...)

enumerate(...)


@@@@@@@@@@@@@@@@@@      DATACLASS      @@@@@@@@@@@@@@@@@@@@@@
- What is dataclass decorator
Introduction - https://www.youtube.com/watch?v=2CvzoNPSOHg

    from dataclasses import dataclass
    @dataclass
    class Person:
        name: str
        age: int
        city: str

    - Decorator takes care of creating following
        - Constructor defining class variables
        - Defining _repr_, _eq_ functions

Parameterized dataclass - https://www.youtube.com/watch?v=Kj8v9mYS0Ik
    @dataclass(cls=None, *, init=True,  repr=True, eq=True, order=False,
                un_safe_hash=False, frozen=False)

Dataclass Field - https://www.youtube.com/watch?v=zyo1PLw4hrE
    field(*, default=MISSING, default_factory=MISSING, init=True, repr=True, hash=None,
            compare=True, metadata=None)
    - This field is assigned to each variable and that modifies

    from dataclasses import dataclass, field
    @dataclass
    class Person:
        name: str
        age: int = field(default=25)
        city: str = field(default_factory=get_common_city)

    - default_factory method must not have any arguments
    - If `init=False` then that variable is not part of __init__() constructor
      when instantiating that class
    - If `repr=False` then that property will not be in object representation
    - If `hash=False` it is not used in computing hash for the class object
    - If `compare=False` it is not used while comparing two objects of same
      class (i.e., __eq__() function ignore this variable)


@@@@@@@@@@@@@@@@@@      EXCEPTION HANDLING   @@@@@@@@@@@@@@@@
test.py
    def x()
        try:
            print(1)
        except:
            print(2)
        else:
            print(3)
        finally:
            print(4)

    x()  <<< what's printed?

- Disassemble above code and see the byte code
    $ python -m dis test.py

- Now, what happens?
test.py
    def x()
        try:
            return 1
        except:
            return 2
        else:
            return 3
        finally:
            return 4

    print(x())  <<< what's printed?


- Exceptions can be a function, or another variable

test.py
    exception_in_variable = Exception
    def exception_fn():
        return Exception

    try:
        ...
    except exception_in_variable:
        ...
    except exception_fn:
        ...


- Use exc_info=True in your logs to get exception information

try:
    ...
except Exception:
    logging.info("an error occurred", exc_info=True)


- DO NOT log MULTIPLE exceptions as follows
    try:
        ...
    except Exception:
        logging.exception("an error occured")
        raise


- Use __repr__ instead of __str__ as follows

    try:
        dict()['nope']
    except KeyError as e:
        logging.error("an error occured: %r", e) ## use %r instead of %s

- Why is __repr__ superior to __str__? Look at this

>>> e = Exception("It broke")
>>>
>>> str(e)
'It broke'
>>>
>>> repr(e)
"Exception('It broke',)"    ## More informative
>>>
>>> str(Exception())
''
>>>
>>> str(ValueError(1))      # We don't know the *type* of error
'1'
>>> str(KeyError(1))
'1'
>>>
>>> repr(KeyError(1))       # Type of error is clear
'KeyError(1,)'
>>> repr(ValueError(1))
'ValueError(1,)'
>>>


Scopes
    try:
        ...
    except Exception as e:
        pass

    print(e)  <<< out of scope


    try:
        ...
    except Exception as e:
        ee = e

    print(ee)  <<< this works


- Don't DOUBLE raise Exception, like this

    try:
        ...
    except Exception:
        raise Exception("This is not good idea. Raises double exception")


- Chaining exceptions
    try:
        1/0
    except ZeroDivisionError as e:
        raise Exception("Bad division") from e

    try:
        1/0
    except Exception:
        raise Exception("Bad division") from None

- Exceptions in Threads
    - Use concurrent.futures library
    - TBD

- Exception hierarchy
    - https://docs.python.org/3/library/exceptions.html#exception-hierarchy




---------------- Talk by Miguel Gringer  ---------------

- LBYL: Look Before You Leap
- EAFTP: Easier to Ask Forgiveness Than Permission

- Don't raise errors. Let higher level scope handle
- If error is unrecoverable, raise an error

- Unrecoverable errors

    WRONG APPROACH

    @app.route('/songs/<id>', methods=['PUT'])
    def update_song(id):
        ...
        try:
            db.session.update(song)
            db.session.commit()
        except SQLAlchemyError as e:
            logger.error("failed to update song %s, %s", song.name, e)
            try:
                db.session.rollback()
            except SQLAlchemyError as e:
                logger.error("error rolling back failed create song, %s", e)
            return 'Internal Server Error', 500
        return '', 204


    RIGHT APPROACH

    @app.route('/songs/<id>', methods=['PUT'])
    def update_song(id):
        db.session.update(song)
        db.session.commit()
        return '', 204


Another Example

    NOT GOOD PRACTICE                               BEST PRACTICE

    def save_customer_to_db(customer):              def save_customer_to_db(customer):
        try:                                            validate_customer(customer)
            validate_customer(customer)                 write_customer(customer)
            write_customer(customer)
        except (ValidationError, IOError) as e:
            # do something here

    if __name__ == "__main__":                      if __name__ == "__main__":
        args = parser.parse_args()                      args = parser.parse_args()
        save_customer_to_db(args.customer)              try:
                                                            save_customer_to_db(args.customer)
                                                        except (ValidationError, IOError) as e:
                                                            logger.exception('Error: ' + str(e))
                                                        except Exception as e:
                                                            logger.exception('Unexpected Error: ' + str(e))

- Merry library (pip install merry). Miguel's own library


https://www.youtube.com/watch?v=V2fGAv2R5j8
Miguel Gringer: https://www.youtube.com/watch?v=8kTlzR4HhWo

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@


Reference:
[1] https://learnpythonthehardway.org/python3/
[2] http://book.pythontips.com/en/latest/
[3] Safari Books Online: REST APIs with Flask and Python (video)
[4] UCSC Extension Class: Python for Programmers

Decorators
[5] http://simeonfranklin.com/blog/2012/jul/1/python-decorators-in-12-steps/
[6] http://thecodeship.com/patterns/guide-to-python-function-decorators/

OOPS
[7] http://www.tutorialspoint.com/python/python_classes_objects.htm
[8] http://www.jeffknupp.com/blog/2014/06/18/improve-your-python-python-classes-and-object-oriented-programming/

Private Methods
[9] http://stackoverflow.com/questions/70528/why-are-pythons-private- methods-not-actually-private
[10] http://effbot.org/pyfaq/tutor-how-do-i-make-public-and-private-attributes- and-methods-in-my-classes.htm

Multiple Inheritance
[11] http://stackoverflow.com/questions/3277367/how-does-pythons-super-work-with-multiple-inheritance
[12] http://en.wikipedia.org/wiki/Multiple_inheritance#The_diamond_problem

Operator Overloading
[13] https://docs.python.org/2/library/operator.html
[14] http://thepythonguru.com/python-operator-overloading/


@@@@@@@@@@@@@@@@@@   Threading and Multiprocessing   @@@@@@@@@@@@@@@@@@@@@@@@@
Basic thread example
---

  import logging
  import threading
  import time

  def thread_function(name):
            # some code

  if __name__ == "__main__":
            ...
   x = threading.Thread(target=thread_function, args=(1,))
   x.start()
            ...

Daemon Threads
---
- If threads are Daemon Threads, then when program ends, it kills the threads
- If threads are NOT Daemon Threads, then when program ends, it waits for
  threads to complete

        x = threading.Thread(target=thread_function, args=(1,), daemon=True)


Working with many threads - 1
---
        import threading

        def thread_function(name):
            # some code

        if __name__ == "__main__":
            threads = list()
            for index in range(3):
                x = threading.Thread(target=thread_function, args=(index,))
                threads.append(x)
                x.start()

            for index, thread in enumerate(threads):
                thread.join()


Working with many threads - 2: ThreadPoolExecutor
---
- Easiest way to run multiple threads
- Present in `concurrent.futures` library
- Use it as context manager using `with` statement


 import concurrent.futures

        def thread_function(name):
            # some code

 if __name__ == "__main__":
     with concurrent.futures.ThreadPoolExecutor(max_workers=3) as executor:
  executor.map(thread_function, range(3))


- The code creates a ThreadPoolExecutor as a context manager, telling it how
  many worker threads it wants in the pool.
- It then uses .map() to step through an iterable of things, in this case
  range(3), passing each one to a thread in the pool.
- The end of the with block causes the ThreadPoolExecutor to do a .join() on
  each of the threads in the pool.
- It is strongly recommended that you use ThreadPoolExecutor as a context
  manager when you can, so that you never forget to .join() the threads.


 import concurrent.futures
 import urllib.request

 URLS = ['http://www.foxnews.com/',
  'http://www.cnn.com/',
  'http://europe.wsj.com/',
  'http://www.bbc.co.uk/',
  'http://some-made-up-domain.com/']

 # Retrieve a single page and report the URL and contents
 def load_url(url, timeout):
     with urllib.request.urlopen(url, timeout=timeout) as conn:
  return conn.read()

 # We can use a with statement to ensure threads are cleaned up promptly
 with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor:

     # Start the load operations and mark each future with its URL
     future_to_url = {executor.submit(load_url, url, 60): url for url in URLS}
     for future in concurrent.futures.as_completed(future_to_url):
  url = future_to_url[future]
  try:
      data = future.result()
  except Exception as exc:
      print('%r generated an exception: %s' % (url, exc))
  else:
      print('%r page is %d bytes' % (url, len(data)))



Basic Synchronization using `Lock`
---
TBD




Reference:
- https://realpython.com/intro-to-python-threading/

@@@@@@@@@@@@@@@@@@   Rich Library @@@@@@@@@@@@@@@@@@@@@@@@@
- Gives tons of helpful info
$ python -m rich

- PRINT: Override python print statement
    from rich import print

- CONSOLE: Console print is even better
    from rich import Console

    console = Console()

    console.print("This is some text")
    console.print("This is some text", style="bold")
    console.print("This is some text", style="bold underline")
    console.print("This is some text", style="bold underline green")
    console.print("This is some text", style="bold underline red on white")

    console.print("[bold]This is [cyan]some[/] text[/]", style="bold underline red on white")


- TEXT: Rich text object
    from rich.text import Text

    text = Text("Hello, World")
    text.stylize("bold magenta", 0, 6)
    console.print(text)

- THEME: Use themes for richer output
    from rich.theme import Theme
    custom_theme = Theme({"success": "green", "error": "bold red"})

    console = Console(theme=custom_theme)

    console.print("Operation successful", style="succes")
    console.print("Operation failed", style="error")
    console.print("Operation [error]failed[/error]") <<< this works too

- EMOJI: Emoji Markup Language can be used, without any additional packages

    console.print(":thumbs_up: File downloaded!")
    console.print(":apple: :bug:")

- CONSOLE.LOG: It's more informative
    console.log("...")

- TRACEBACKS: Very useful info is added in Tracebacks
    console.log("...", log_locals=True) <<< this logs local variables

    from rich.traceback import install
    install()

    Now, any traceback has awesome output


- SAVE HTML: Saves output to HTML

    console = Console(record=True)  <<< record must be True, to save to HTML
    try:
        ...
    except:
        console.print_exception()

    console.save_html("demo.html")  <<< saves in an html


- TABLE: Save output in tables

    from rich.table import Table

    table = Table(title="Star Wars Movies")

    table.add_column("Released", style="cyan")
    table.add_column("Title", style="magenta")
    table.add_column("Box Office", justify="left", style="green")

    table.add_row("Dec 20, 2019", "Star Wars: The Rise of Skywalker", "$952,110,690")
    table.add_row("May 25, 2018", "Solo: A Star Wars Story", "$393,151,347")
    table.add_row("Dec 15, 2017", "Star Wars Ep. VIII: The Last Jedi", "$1, 332, 539, 889")

    console = Console()
    console.print(table)


- MARKDOWN:
    from rich.markdown import Markdown

    MARKDOWN = """
    # This is h1

    Rich can do a pretty *decent* job of rendering markdown.

    1. This is a list item.
    2. This is another item.

    """

    md = Markdown(MARKDOWN)
    console = Console()
    console.print(md)


    python -m rich.markdown <FILENAME.md>


- PROGRESSBAR

    from rich.progress import track

    from i in track(range(10, description="Processing ..."):
        print(f"working {i}")
        time.sleep(1)


Reference:
[1] https://www.youtube.com/watch?v=4zbehnz-8QU&t=6s


@@@@@@@@@@@@@@@@@@   strftime cheat sheet  @@@@@@@@@@@@@@@@@@@@
datetime.now().strftime('%y')           '22'
datetime.now().strftime('%Y')           '2022'
datetime.now().strftime('%Y-%m')        '2022-02'
datetime.now().strftime('%Y-%m-%d')     '2022-02-16'
datetime.now().strftime('%Y-%h-%d')     '2022-Feb-16'

datetime.now().strftime('%Y-%m-%d-%H')          '2022-02-16-10'
datetime.now().strftime('%Y-%m-%d-%H:%M:%S')    '2022-02-16-10:32:36'
datetime.now().strftime('%Y-%m-%d-%H:%M:%S%p')  '2022-02-16-10:32:36AM'
datetime.now().strftime('%Y-%m-%d-%H:%M:%S:%f')  '2022-02-16-10:32:36:01995800'


datetime.now().strftime('%s')           '1645036702'
datetime.now().strftime('%c')           'Wed Feb 16 10:39:24 2022'
datetime.now().strftime('%x')           '02/16/22'
datetime.now().strftime('%D')           '02/16/22'
datetime.now().strftime('%X')           '10:42:23'



Reference:
[1] https://strftime.org/
[2] https://docs.python.org/3/library/datetime.html#strftime-and-strptime-format-codes

@@@@@@@@@@@@@@@@@@   Argparse  Notes       @@@@@@@@@@@@@@@@@@@@

    # Import the library
    import argparse

    # Create the parser
    parser = argparse.ArgumentParser()

    # Add an argument
    parser.add_argument('--name', type=str, required=True)
        # type indicates ... type
        # 'required' is boolean type

    # Parse the arguments
    args = parser.parse_args()

The add_argument() method
-------------------------
ArgumentParser.add_argument(name or flags...[, action][, nargs][, const][, default][, type][, choices][, required][, help][, metavar][, dest])
    Define how a single command-line argument should be parsed. Each parameter
    has its own detailed description below, but in short they are:

name or flags - Either a name or a list of option strings
    e.g. foo or -f, --foo.

action - The basic type of action to be taken when this argument is encountered at the command line.
    action='store_true' means default=False
    action='store_false' means default=True
    - If you want only a flag without value, use above
    - https://docs.python.org/dev/library/argparse.html#action
    - https://stackoverflow.com/a/36710639

nargs - The number of command-line arguments that should be consumed.
const - A constant value required by some action and nargs selections.
default - The value produced if the argument is absent from the command line and if it is absent from the namespace object.
type - The type to which the command-line argument should be converted.
choices - A container of the allowable values for the argument.
required - Whether or not the command-line option may be omitted (optionals only).
help - A brief description of what the argument does.
metavar - A name for the argument in usage messages.
dest - The name of the attribute to be added to the object returned by parse_args().



Mutually Exclusive Group
========================

mydaemon.py
    parser = argparse.ArgumentParser(prog='mydaemon')
    action = parser.add_mutually_exclusive_group(required=True)
    action.add_argument('--start', action='store_true', help='Starts %(prog)s daemon')
    action.add_argument('--stop', action='store_true', help='Stops %(prog)s daemon')
    action.add_argument('--restart', action='store_true', help='Restarts %(prog)s daemon')


usage: mydaemon [-h] (--start | --stop | --restart)

optional arguments:
  -h, --help  show this help message and exit
  --start     Starts mydaemon daemon
  --stop      Stops mydaemon daemon
  --restart   Restarts mydaemon daemon



Sub Parsers
===========
TODO

[1] https://towardsdatascience.com/a-simple-guide-to-command-line-arguments-with-argparse-6824c30ab1c3
[2] https://docs.python.org/3/library/argparse.html#the-add-argument-method

==================    NOTES   FROM   PYTHON   TALKS   ===============
- Look these up:
    reversed(), enumerate(), zip(), izip(), sorted()

[1] Transforming Code in to Beautiful, Idiomatic Python - Raymond Hettinger
https://www.youtube.com/watch?v=anrOzOapJ2E&list=WL&index=23&t=19s



==================    DUNDER/MAGIC/SPECIAL METHODS IN PYTHON     ========


Dunder method     Usage / Needed for
__init__         Initialise object
      Link: https://docs.python.org/3/reference/datamodel.html#object.__init__
__new__           Create object
      Link: https://docs.python.org/3/reference/datamodel.html#object.__new__
__del__           Destroy object
      Link: https://docs.python.org/3/reference/datamodel.html#object.__del__
__repr__         Compute “official” string representation / repr(obj)
      Link: https://docs.python.org/3/reference/datamodel.html#object.__repr__
            https://mathspp.com/blog/pydonts/str-and-repr
__str__           Pretty print object / str(obj) / print(obj)
      Link: https://docs.python.org/3/reference/datamodel.html#object.__str__
            https://mathspp.com/blog/pydonts/str-and-repr
__bytes__         bytes(obj)
      Link: https://docs.python.org/3/reference/datamodel.html#object.__bytes__
__format__       Custom string formatting
      Link: https://docs.python.org/3/reference/datamodel.html#object.__format__
            https://mathspp.com/blog/pydonts/string-formatting-comparison#custom-formatting
__lt__           obj < ...
      Link: https://docs.python.org/3/reference/datamodel.html#object.__lt__
__le__           obj <= ...
      Link: https://docs.python.org/3/reference/datamodel.html#object.__le__
__eq__           obj == ...
      Link: https://docs.python.org/3/reference/datamodel.html#object.__eq__
__ne__           obj != ...
      Link: https://docs.python.org/3/reference/datamodel.html#object.__ne__
__gt__           obj > ...
      Link: https://docs.python.org/3/reference/datamodel.html#object.__gt__
__ge__           obj >= ...
      Link: https://docs.python.org/3/reference/datamodel.html#object.__ge__
__hash__         hash(obj) / object as dictionary key
      Link: https://docs.python.org/3/reference/datamodel.html#object.__hash__
__bool__         bool(obj) / define Truthy/Falsy value of object
      Link:
__getattr__       Fallback for attribute access
      Link:
__getattribute__ Implement attribute access: obj.name
      Link:
__setattr__       Set attribute values: obj.name = value
      Link:
__delattr__       Delete attribute: del obj.name
      Link:
__dir__           dir(obj)
      Link:
__get__           Attribute access in descriptor
      Link:
__set__           Set attribute in descriptor
      Link:
__delete__       Attribute deletion in descriptor
      Link:
__init_subclass__ Initialise subclass
      Link:
__set_name__     Owner class assignment callback
      Link:
__instancecheck__ isinstance(obj, ...)
      Link:
__subclasscheck__ issubclass(obj, ...)
      Link:
__class_getitem__ Emulate generic types
      Link:
__call__         Emulate callables / obj(*args, **kwargs)
      Link:
__len__           len(obj)
      Link:
__length_hint__   Estimate length for optimisation purposes
      Link:
__getitem__       Access obj[key]
      Link:
__setitem__       obj[key] = ... or `obj[]
      Link:
__delitem__       del obj[key]
      Link:
__missing__       Handle missing keys in dict subclasses
      Link:
__iter__         iter(obj) / for ... in obj (iterating over)
      Link:
__reversed__     reverse(obj)
      Link:
__contains__     ... in obj (membership test)
      Link:
__add__           obj + ...
      Link:
__radd__         ... + obj
      Link:
__iadd__         obj += ...
      Link:
__sub__ 2 3       obj - ...
      Link:
__matmul__ 2 3   obj @ ...
      Link:
__mul__ 2 3       obj * ...
      Link:
__truediv__ 2 3   obj / ...
      Link:
__floordiv__ 2 3 obj // ...
      Link:
__mod__ 2 3       obj % ...
      Link:
__divmod__ 2     divmod(obj, ...)
      Link:
__pow__ 2 3       obj ** ...
      Link:
__lshift__ 2 3   obj << ...
      Link:
__rshift__ 2 3   obj >> ...
      Link:
__and__ 2 3       obj & ...
      Link:
__xor__ 2 3       obj ^ ...
      Link:
__or__ 2 3       obj | ...
      Link:
__neg__           -obj (unary)
      Link:
__pos__           +obj (unary)
      Link:
__abs__           abs(obj)
      Link:
__invert__       ~obj (unary)
      Link:
__complex__       complex(obj)
      Link:
__int__           int(obj)
      Link:
__float__         float(obj)
      Link:
__index__         Losslessly convert to integer
      Link:
__round__         round(obj)
      Link:
__trunc__         math.trunc(obj)
      Link:
__floor__         math.floor(obj)
      Link:
__ceil__          math.ceil(obj)
      Link:
__enter__         with obj (enter context manager)
      Link:
__exit__          with obj (exit context manager)
      Link:
__await__         Implement awaitable objects
      Link:
__aiter__         aiter(obj)
      Link:
__anext__         anext(obj)
      Link:
__aenter__       async with obj (enter async context manager)
      Link:
__aexit__         async with obj (exit async context manager)
      Link:

References:
[1] https://mathspp.com/blog/pydonts/dunder-methods
[2] https://holycoders.com/python-dunder-special-methods/

==================    Python Logging/Logger        ===================

Simple Example:
    https://docs.python.org/3/howto/logging.html#a-simple-example

Logging to a file:
    https://docs.python.org/3/howto/logging.html#logging-to-a-file

Logging from multiple modules:
    https://docs.python.org/3/howto/logging.html#logging-from-multiple-modules

Changing format of displayed messages:
    https://docs.python.org/3/howto/logging.html#logging-from-multiple-modules

Displaying Date/Time of Messages:
    https://docs.python.org/3/howto/logging.html#displaying-the-date-time-in-messages

Advanced Logging Tutorial:
    https://docs.python.org/3/howto/logging.html#advanced-logging-tutorial



[1] https://docs.python.org/3/howto/logging.html#logging-basic-tutorial
[2] https://docs.python.org/3/howto/logging-cookbook.html

==================    Python AsyncIO    ===================

import asyncio

async def func():
    print("A")
    await asyncio.sleep(1)
    print("B")

async.run(fun())

---------------

import asyncio

async def func():
    task = asyncio.create_task(second_func())
    print("A")
    print("B")
    return_value = await task

async def second_func():
    print("1")
    await asyncio.sleep(1)
    print("2")
    return 10

async.run(fun())

---------------


References:


##############################    Faker Library   ######################
from faker import Faker
fake = Faker()

    fake.address()
    fake.country()
    fake.currency()
    fake.currency_name(), fake.currency_code()
    fake.email()
    fake.job()
    fake.latitude(), fake.longitude()
    fake.phone_number()
    fake.name()
    fake.profile()
    fake.sentence()
    fake.text()
    fake.url()
    fake.word(), fake.words()


#'hi_IN' changed the language to Hindi
fake = Faker('hi_IN')

- Lot more info here:
    https://zetcode.com/python/faker/


##############################    Hypothesis Package   #################
- Hypothesis is a property based testing package with roots in Functional
  Programming. Inspired by, QuickCheck framework for Haskell.


ascii.py
    from functools import reduce

    def to_ascii_codes(inp: str) -> list[int]:
 return [ord(c) for c in inp]

    def from_ascii_codes(inp: list[int]) -> str:
 return reduce(lambda x, y: x + chr(y), inp, "")


test_ascii.py
    from hypothesis


##############################    xml.etree.ElementTree      #################
- The xml.etree.ElementTree (ET in short) module implements a simple and
  efficient API for parsing and creating XML data.
- XML is an inherently hierarchical data format, and the most natural way to
  represent it is with a tree.
- ET has two classes for this purpose
    - `ElementTree` represents the whole XML document as a tree, and
    - `Element` represents a single node in this tree.
    - Interactions with the whole document (reading and writing to/from files)
      are usually done on the `ElementTree` level.
    - Interactions with a single XML element and its sub-elements are done on
      the `Element` level.


Parsing XML
===========
- We’ll be using the following XML document as the sample data for this section:
 <?xml version="1.0"?>
 <data>
     <country name="Liechtenstein">
  <rank>1</rank>
  <year>2008</year>
  <gdppc>141100</gdppc>
  <neighbor name="Austria" direction="E"/>
  <neighbor name="Switzerland" direction="W"/>
     </country>
     <country name="Singapore">
  <rank>4</rank>
  <year>2011</year>
  <gdppc>59900</gdppc>
  <neighbor name="Malaysia" direction="N"/>
     </country>
     <country name="Panama">
  <rank>68</rank>
  <year>2011</year>
  <gdppc>13600</gdppc>
  <neighbor name="Costa Rica" direction="W"/>
  <neighbor name="Colombia" direction="E"/>
     </country>
 </data>


- We can import this data by reading from a file:

 import xml.etree.ElementTree as ET
 tree = ET.parse('country_data.xml')
 root = tree.getroot()


- Or directly from a string:
 root = ET.fromstring(country_data_as_string)

- `fromstring()` parses XML from a string directly into an `Element`, which is
  the root element of the parsed tree. Other parsing functions may create an
  `ElementTree`. Check the documentation to be sure.

- As an `Element`, root has a tag and a dictionary of attributes:

 >>> root.tag
 'data'
 >>> root.attrib
 {}

- It also has children nodes over which we can iterate:

 >>> for child in root:
 ... print(child.tag, child.attrib)
 ...
 country {'name': 'Liechtenstein'}
 country {'name': 'Singapore'}
 country {'name': 'Panama'}

- Children are nested, and we can access specific child nodes by index:
 >>> root[0][1].text
 '2008'


- NOTE: Not all elements of the XML input will end up as elements of the parsed
  tree. Currently, this module skips over any XML comments, processing
  instructions, and document type declarations in the input. Read documentation
  for more details.


Finding interesting elements
============================
- `Element` has some useful methods that help iterate recursively over all the
  sub-tree below it (its children, their children, and so on).
-  For example, `Element.iter()`:

 >>> for neighbor in root.iter('neighbor'):
 ... print(neighbor.attrib)
 ...
 {'name': 'Austria', 'direction': 'E'}
 {'name': 'Switzerland', 'direction': 'W'}
 {'name': 'Malaysia', 'direction': 'N'}
 {'name': 'Costa Rica', 'direction': 'W'}
 {'name': 'Colombia', 'direction': 'E'}

- `Element.iter(tag=None)`
    - Creates a tree iterator with the current element as the root.
    - The iterator iterates over this element and all elements below it, in
      document (depth first) order.
    - If tag is not None or '*', only elements whose tag equals tag are
      returned from the iterator.
    - If the tree structure is modified during iteration, the result is
      undefined.


- `Element.findall(match, namespace=None)` finds only elements with a tag which
  are direct children of the current element.
    - Finds all matching subelements, by tag name or path. Returns a list
      containing all matching elements in document order.

- `Element.find(match, default=None, namespace=None)` finds the first child
  with a particular tag.
    - Finds the first subelement matching match. match may be a tag name or a
      path. Returns an element instance or None.

- `Element.text` accesses the element’s text content after start tag.
- `Element.tail` accesses the element’s text content after end tag.

 - These attributes can be used to hold additional data associated with
   the element.
 - If the element is created from an XML file, the `text` attribute
   holds either the text between the element’s start tag and its first
   child or end tag, or `None`, and the `tail` attribute holds either
   the text between the element’s end tag and the next tag, or `None`.
 - For the XML data

  <a><b>1<c>2<d/>3</c></b>4</a>

  the `a` element has `None` for both `text` and `tail`
  attributes, the `b` element has text "1" and tail "4", the `c`
  element has text "2" and tail None, and the `d` element has
  text None and tail "3".

- `Element.attrib`:
    - A dictionary containing the element’s attributes.
    - The following dictionary-like methods work on the element attributes.
 - clear()
 - get(key, default=None)
 - items()
 - keys()
 - set(key, value)
 - append(subelement)
 - extend(subelements)
 - find(match, namespaces=None)
 - findall(match, namespaces=None)
 - findtext(match, default=None, namespaces=None)
 - insert(index, subelement)
 - iter(tag=None)
 - iterfind(match, namespace=None)
 - remove()


- `Element.get()` accesses the element’s attributes.

 >>> for country in root.findall('country'):
 ... rank = country.find('rank').text
 ... name = country.get('name')
 ... print(name, rank)
 ...
 Liechtenstein 1
 Singapore 4
 Panama 68

- More sophisticated specification of which elements to look for is possible by using `XPath`.




- ElementTreee Objects
======================
class xml.etree.ElementTree.ElementTree(element=None, file=None)
    - ElementTree wrapper class. This class represents an entire element
      hierarchy, and adds some extra support for serialization to and from
      standard XML.

    - `element` is the root element. The tree is initialized with the
      contents of the XML `file` if given.

_setroot(element)
    - Replaces the root element for this tree. This discards the current
      contents of the tree, and replaces it with the given element. Use with
      care. `element` is an element instance.

find(match, namespaces=None)
    - Same as `Element.find()`, starting at the root of the tree.

findall(match, namespaces=None)
    - Same as `Element.findall()`, starting at the root of the tree.

findtext(match, default=None, namespaces=None)
    - Same as Element.findtext(), starting at the root of the tree.

getroot()
    - Returns the root element for this tree.

iter(tag=None)
    - Creates and returns a tree iterator for the root element. The iterator
      loops over all elements in this tree, in section order. tag is the tag to
      look for (default is to return all elements).

iterfind(match, namespaces=None)
    - Same as Element.iterfind(), starting at the root of the tree.

Reference:
[1] https://docs.python.org/3/library/xml.etree.elementtree.html