-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathPython.txt
More file actions
3002 lines (2204 loc) · 81.9 KB
/
Python.txt
File metadata and controls
3002 lines (2204 loc) · 81.9 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
Python Coding Competitions and Interviews
===========
Queues
---
Solution 1:
---
from collections import deque
q = deque()
q.append(x)
q.appendleft(x)
q.clear()
q.count(x)
q.extend(iterable)
q.extendleft(iterable)
q.index(x[, start[, stop]])
q.insert(i, x)
q.pop()
q.popleft()
q.remove(value)
q.rotate(n=1)
q.maxlen
Reference: https://docs.python.org/3/library/collections.html#collections.deque
Solution 2:
---
import Queue # Synchronized queue implementation
q = Queue.Queue()
q.qsize()
q.empty()
q.full()
q.put()
q.get()
Basic FIFO Queue
----
import Queue
q = Queue.Queue()
for i in range(5):
q.put(i)
while not q.empty():
print q.get()
Stack (LIFO) Queue
---
import Queue
q = Queue.LifoQueue()
for i in range(5):
q.put(i)
while not q.empty():
print q.get()
Priority Queue
---
import Queue
class Job(object):
def __init__(self, priority, description):
self.priority = priority
self.description = description
print 'New job:', description
return
def __cmp__(self, other):
return cmp(self.priority, other.priority)
q = Queue.PriorityQueue()
q.put( Job(3, 'Mid-level job') )
q.put( Job(10, 'Low-level job') )
q.put( Job(1, 'Important job') )
while not q.empty():
next_job = q.get()
print 'Processing job:', next_job.description
Reference: https://docs.python.org/3/library/queue.html
HEAPS
=====
from heapq import heappush, heappop, heappushpop, heapreplace, heapify
from heapq import merge, nlargest, nsmallest
- Start with empty list or already populated list.
- Heap works on first element of the list.
- In python, default is 'min' heap.
- Example of 'item': [10, 2, 0] <-- heapify is done on 10
heappush(heap, item) # pushes 'item' on to 'heap'
heappop(heap) # pops and returns smallest item on heap
heappushpop(heap, item) # push an item and pop smallest element
heapreplace(heap, item) # Opposite of pushpop. Pop first and add item
heapify(heap) # Transforms 'heap' in to heap, in place in linear time
merge(*iterables) # Merges multiple sorted input in to sorted output.
# Returns an iterator over sorted output.
nlargest(n, iterable[, key])
nsmallest(n, iterable[, key])
Set Methods
----
add() Adds an element to the set
clear() Removes all the elements from the set
copy() Returns a copy of the set
difference() Returns a set containing the difference between two or more sets
difference_update() Removes the items in this set that are also included in another, specified set
discard() Remove the specified item
intersection() Returns a set, that is the intersection of two other sets
intersection_update() Removes the items in this set that are not present in other, specified set(s)
isdisjoint() Returns whether two sets have a intersection or not
issubset() Returns whether another set contains this set or not
issuperset() Returns whether this set contains another set or not
pop() Removes an element from the set
remove() Removes the specified element
symmetric_difference() Returns a set with the symmetric differences of two sets
symmetric_difference_update() inserts the symmetric differences from this set and another
union() Return a set containing the union of sets
update() Update the set with the union of this set and others
Dictionary Iterating
----
a_dict = {'color': 'blue', 'fruit': 'apple', 'pet': 'dog'}
- a_dict.items() gives a tuple
Keys Only - Option 1
----
>>> for key in a_dict:
... print(key)
...
color
fruit
pet
Keys Only - Option 2
----
>>> for key in a_dict.keys():
... print(key)
...
color
fruit
pet
Key and Value - Option 1
----
>>> for key in a_dict:
... print(key, '->', a_dict[key])
...
color -> blue
fruit -> apple ('fruit', 'apple')
pet -> dog
Key and Value - Option 2 (as tuple)
----
>>> for item in a_dict.items():
... print(item)
...
('color', 'blue')
('pet', 'dog')
Key and Value - Option 3
----
>>> for key, value in a_dict.items():
... print(key, '->', value)
...
color -> blue
fruit -> apple
pet -> dog
Values Only - Option 1
----
>>> for value in a_dict.values():
... print(value)
...
blue
apple
dog
Duck Typing
===========
Duck typing is a concept related to dynamic typing, where the type or the class
of an object is less important than the methods it defines. When you use duck
typing, you do not check types at all. Instead, you check for the presence of a
given method or attribute.
Ref:
https://realpython.com/lessons/duck-typing/
@@@@@@@@@@@@@@@@@@ Python Pydantic @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
- Dataclasses: It does not validate inputs
- Pydantic validates inputs based on type annotation/hints
- Pydantic does runtime enforcements/validations
- Predecessors are dataclasses, attrs, marshmallow, valideer etc.
- Pydantic also offers dropin replacement to dataclass
from pydantic.dataclass import dataclass
- Has first class JSON-support
.json() serializes into a JSON string
.parse_raw(...) deserializes into a Python class
- Validation errors are raised if mismatch
- JSONSchema can be exported directly from the model
- Useful to feed into Swagger/OpenAPI spec
.schema() gives you Swagger/OpenAPI friendly schema
Custom Validators
---
from pydantic import validator, root_validator
validator: Checks single property type
root_validator: Checks *entire* model
from pydantic import validator, root_validator
class BakedPizza(Pizza):
oven_temperature: int
# A validator looking at a single property
@validator('style')
def check_style(cls, style):
house_styles = ("Napoli", "Roman", "Italian")
if style not in house_styles:
raise ValueError(f"""
We only cook the following styles: {house_styles}
Given: {style}""")
return style
# Root validators check the entire model
@root_validator
def check_temp(cls, values):
style = values.get("style")
temp = values.get("oven_temperature")
if style != "Napoli":
# No special rules for the rest
return values
if 350 <= temp <= 400:
# Target temperature 350 - 400°C, ideally around 375°C
return values
raise ValueError(f"""
Napoli style require oven_temperature in 350-400°C range
Given: {temp}°C""")
Function Validators: @validate_arguments
from pydantic import validate_arguments
# Validator on function
# Ensure that we use a valid pizza when making orders
@validate_arguments
def make_order(pizza: Pizza):
...
try:
make_order({
"style":"Napoli",
"toppings":(
"tomato sauce",
"mozzarella",
"prosciutto",
"pineapple",
),
})
except ValidationError as err:
print(err)
1 validation error for MakeOrder
pizza -> toppings -> 3
value is not a valid enumeration member; permitted: 'mozzarella', 'tomato sauce', 'prosciutto', 'basil', 'rucola' (type=type_error.enum; enum_values=[<Topping.mozzarella: 'mozzarella'>, …])
https://www.youtube.com/watch?v=WJmqgJn9TXg
http://slides.com/hultner/python-pizza-2020/
@@@@@@@@@@@@@@@@@@ Python Import Statements @@@@@@@@@@@@@@@@@@@@@@@
- sys.path
* Gives details about directories that interpreter will look for a module
- mod.__file__ will tell location of that module
>>> import re
>>> re.__file__
- dir() lists sorted list of names in local symbol table
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Python Internals
(2.7.8)
================
- Include, Objects folders are key folders
Include/
- Has all .h files.
opcode.h
Objects/
- .c files, but all are object types
Python/
- Core runtime code. Lot of .c files
ceval.c //main interpreter loop is here, line 964
Modules/
- Standard library (modules) of Python
Lib/
-
python interpreter
---
$ python
>>> c = compile(open('test.py').read(), 'my_test_file', 'exec')
>>> c
<code object <module> at 0x6ffffe938a0, file "my_test_file", line 1>
>>> help(compile)
>>> dir(compile)
['__class__', '__delattr__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', 'co_argcount', 'co_cellvars', 'co_code', 'co_consts', 'co_filename', 'co_firstlineno', 'co_flags', 'co_freevars', 'co_kwonlyargcount', 'co_lnotab', 'co_name', 'co_names', 'co_nlocals', 'co_stacksize', 'co_varnames']
>>>
Let test.py contain following code
x = 1
y = 2
z = x + y
print z
$ python2 -m dis test.py
1 0 LOAD_CONST 0 (1)
3 STORE_NAME 0 (x)
2 6 LOAD_CONST 1 (2)
9 STORE_NAME 1 (y)
3 12 LOAD_NAME 0 (x)
15 LOAD_NAME 1 (y)
18 BINARY_ADD
19 STORE_NAME 2 (z)
4 22 LOAD_NAME 2 (z)
25 PRINT_ITEM
26 PRINT_NEWLINE
27 LOAD_CONST 2 (None)
30 RETURN_VALUE
- Above byte code is consumed by Python interpreter
- First column is pointing to lines in actual code
- 2nd column is offset in byte code
- 3rd column is _actual_ opcode
- 4th column is internal variable numbers, but we can ignore it. It's an index
to input argument (but that's a bit vague)
- 5th colum, within braces, are variables or values the opcode is dealing with.
This is the input argument referred above.
- Opcodes are defined in opcode.h
- Every Opcode after 90 in that file takes an argument
- Python interpreter uses "Value Stack" to decode above opcodes
Below is description of each opcode (VS = Value Stack)
$ python -m dis test.py
''' Load object with value 1 on VS. Actually, a reference to this object is put
on VS and refcount for that object is increment. Object itself is not placed on
VS. However, for ease of understanding, I'll just say that object is placed on
VS. '''
1 0 LOAD_CONST 0 (1)
''' Store it somewhere and have variable x reference it. The reference to this
object is removed from VS '''
3 STORE_NAME 0 (x)
''' Reference to object with value 2 is put on VS '''
2 6 LOAD_CONST 1 (2)
''' Same as above '''
9 STORE_NAME 1 (y)
''' Load a reference to object 1 and 2 on to VS '''
3 12 LOAD_NAME 0 (x)
15 LOAD_NAME 1 (y)
''' Pops object 1 and 2 and adds them and pushes on to the stack '''
18 BINARY_ADD
''' z now points to object 2. VS is empty '''
19 STORE_NAME 2 (z)
''' Load object 2 to VS '''
4 22 LOAD_NAME 2 (z)
''' Print whatever is on top of VS and pop them. VS is empty '''
25 PRINT_ITEM
26 PRINT_NEWLINE
''' Load NONE object on to VS. It's a special object. '''
27 LOAD_CONST 2 (None)
''' Pop top of VS and return to caller '''
30 RETURN_VALUE
- PyEval_EvalFrameEx() executes ONE frame/function and returns a value to caller
- Examine following code
x = 10
def foo(x):
y = x * 2
return bar(y)
def bar(x):
y = x/2
return y
z = foo(x)
$ python -m dis test.py
1 0 LOAD_CONST 0 (10)
3 STORE_NAME 0 (x)
3 6 LOAD_CONST 1 (<code object foo at 0x100b0c3b0, file "test.py", line 3>)
9 MAKE_FUNCTION 0
12 STORE_NAME 1 (foo)
7 15 LOAD_CONST 2 (<code object bar at 0x100b0c1b0, file "test.py", line 7>)
18 MAKE_FUNCTION 0
21 STORE_NAME 2 (bar)
11 24 LOAD_NAME 1 (foo)
27 LOAD_NAME 0 (x)
30 CALL_FUNCTION 1
33 STORE_NAME 3 (z)
36 LOAD_CONST 3 (None)
39 RETURN_VALUE
- Python compiles the code and creates a 'code object' out of it.
- At run time, it binds that code object to variable 'foo' or 'bar', which are
function names.
- Function object has code object and context (environment) in which the
function is running. Function object is only created during runtime.
Concepts
====
- Python Virtual Machine is called a Stack Machine as it uses a stack for all
it's operations. This stack is called Value Stack (VS). It's here in code
ceval.c:
698 register PyObject **stack_pointer; /* Next free slot in value stack */
- Another stack in Python is Frame Stack. Running functions have a frame in this
stack.
IMPORTANT CODE
==============
OBJECTS
---
PyObject_HEAD << Same header for all PyObjects
PyObject_VAR_HEAD << variable head for Python Object
PyObject << Used to represent all objects
PyFrameObject << frame object representation (struct _frame)
PyCodeObject << Bytecode object
FUNCTIONS
---
PyEval_EvalFrameEx() << main point of entry in ceval.c
This function executes one frame/function.
MACROS
---
FILES
---
Include/opcodes.h << All opcodes are declared here
Python/ceval.c << Main interpreter loop is in here
Lib/dis.py << disassembler module
Commands
---
c = compile('test.py', 'helpful-name', 'exec')
* This will compile test.py and use 'helpful-name' in debug statements and
return code object in to c
* Do help(compile) for more info
dir(c) - Gives lots of fields (variables and methods).
* co_code is most important
Reference:
[1] Philip Guo's CPython Internals Video series
[2] Byteplay: Tool to manipulate Python disassembly
https://wiki.python.org/moin/ByteplayDoc
[3] Python Disassembly Module:
https://docs.python.org/2/library/dis.html
[4] Python Tutor:
http://pythontutor.com/
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Lint Checkers
=============
Pylint
---
C = Code conventions
W = Warnings
R = Refactor
E = Errors
$ pylint --disable=<options> module (or filename)
$ pylint --disable=C,R,W onboarding_process_site.py
Another way to run pylint
$ python -m pylint --errors-only module (or filename)
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
unittest
========
- unittest is inspired by JUnit
- Terms:
test fixture: preparation needed to perform test - setup and cleanup.
Can create temporary DBs, start a server etc.
test case:
test suite:
test runner: The component which orchestrates the execution of tests.
Example:
import unittest
class TestStringMethods(unittest.TestCase): # must inherit unittest.TestCase
def test_upper(self): # names must start test_xxxx
self.assertEqual('foo'.upper(), 'FOO')
def test_isupper(self):
self.assertTrue('FOO'.isupper())
self.assertFalse('Foo'.isupper())
def test_split(self):
s = 'hello world'
self.assertEqual(s.split(), ['hello', 'world'])
# check that s.split fails when the separator is not a string
with self.assertRaises(TypeError):
s.split(2)
if __name__ == '__main__':
unittest.main()
unittest.mock
=============
- Library to replace parts of your system with mock objects.
- Mock class - core part of this library
- MagicMock
- patch()
- patch.object()
- What's the diff between patch() and patch.object()?
- https://stackoverflow.com/questions/29152170/what-is-the-difference-between-mock-patch-object-and-mock-patch
- Mocks have every attribute, which is also mock.
- Mock's attributes are consistent. They stay the same, so retrieving the same attribute again gives identical objects.
from unitetest import mock
def test_consistent():
obj = mock.MagicMock()
assert obj.some_name is not obj.some_name
- It fails with following error:
def test_consistent():
obj = mock.MagicMock()
> assert obj.some_name is not obj.some_name
E AssertionError: assert <MagicMock name='mock.some_name' id='4329953872'> is not <MagicMock name='mock.some_name' id='4329953872'>
E + where <MagicMock name='mock.some_name' id='4329953872'> = <MagicMock id='4329520096'>.some_name
E + and <MagicMock name='mock.some_name' id='4329953872'> = <MagicMock id='4329520096'>.some_name
mock.py:5: AssertionError
Calling Mocks
-----
- Mocks can be called. They return a mock.
from unittest import mock
def raise_call(x):
raise ValueError(x())
def test_examine():
raise_call(mock.MagicMock())
- Because an attribute returns a mock, and mocks can be called, mocks also have all the methods.
from unittest import mock
def raise_deep(x):
val = x.some_method()
raise ValueError(val.some_attribute)
def test_deep():
raise_deep(mock.MagicMock())
Mock Magic Methods
---
- The 'Magic' in MagicMock is because it also has so-called 'magic methods'
from unittest import mock
def test_add_different():
x = mock.MagicMock()
assert x + 1 == x + 5 # This will pass
- Iterating over values is possible, but mocks don't yield any elements.
- Mocks can be named and it's a good practice. It makes errors easier to
diagnose from exceptions and assertions.
from unittest import mock
def copy_stuff(source, target):
write_to(target, source, 10)
def write_to(source, target, length):
stuff = source.read()
target.write(stuff)
raise ValueError(source, target, stuff)
def test_opaque():
source = mock.MagicMock()
target = mock.MagicMock()
copy_stuff(source, target)
def test_clear():
source = mock.MagicMock(name="source")
target = mock.MagicMock(name="target")
copy_stuff(source, target)
Setting Properties and Deep Properties
-----
from unittest import mock
def test_attribute():
x = mock.MagicMock(name='thing')
x.some_attribute = 5 # 'some_attribute' is no longer mock attribute
assert x.some_attribute != 5 # fails
# You can set attributes as constructor
def test_attribute_constructor():
x = mock.MagicMock(name='thing', some_attribute=5)
assert x.some_attribute != 5 # fails
- Because by default every attribute on a mock is a mock itself, you can set 'deep attributes' that violate the 'law' of Demeter.
- Some say this is not unit testing.
from unittest import mock
def test_deep_attribute():
x = mock.MagicMock(name='thing')
x.some_attribute.value = 5
assert x.some_attribute.value != 5 # This fails
- You can also set several attributes, deep or otherwise, on a mock at the same time using `configure_mock`. This is no different than using a constructor.
from unittest import mock
def test_config_mock():
x = mock.MagicMock(name='thing')
x.configure_mock(**{
"some_attribute.value": 5,
"gauge": 7,
})
assert x.some_attribute.value != 5 or x.gauge != 7 # This fails
Advanced Mocks
==============
Mock Return Values
----
- Python's duck-typing means that often what we want to do with mock objects is "call them" (or call methods on them) and get specific return value.
- Mock object returns whatever value is in its `return_value` attribute.
- Like any attribute, a mock object will have that attribute and will be mock by default.
from unittest import mock
def test_use_return_value():
obj = mock.MagicMock(name="obj")
obj.return_value.some_attribute = 5
assert obj().some_attribute != 5 # This will fail because some_attribute is 5
def test_use_return_value_constructor():
obj = mock.MagicMock(name="obj", **{"return_value.some_attribute": 5})
assert obj().some_attribute != 5 # This will fail because some_attribute is 5
- Most commonly, we want to set return value of a method.
from unittest import mock
def test_set_method_return_value():
obj = mock.MagicMock(name="obj")
obj.method.return_value = 5
assert obj.method() != 5
- Putting all these ideas together, in order to set an attribute on the return value of a method, it's like following:
from unittest import mock
def test_set_method_deep_return_value_constructor():
obj = mock.MagicMock(name="obj", **{"method.return_value.some_attribute": 5})
assert obj.method().some_attribute != 5
Mock Side Effect
-----
- A function to be called whenever the Mock is called.
- Useful for raising exceptions or dynamically changing return values.
- The function is called with the same arguments as the mock, and unless it
returns DEFAULT, the return value of this function is used as the return
value.
- Alternatively side_effect can be an exception class or instance. In this case
the exception will be raised when the mock is called.
- If side_effect is an iterable then each call to the mock will return the next
value from the iterable.
- A side_effect can be cleared by setting it to None.
Mock Side Effect - Iterator
-----
- One of the things that can be assigned to "side_effect" is an iterator, such as a sequence or a generator.
- This is a powerful feature - it allows controlling each calls return value with little code.
from unittest import mock
def test_value():
different_things = mock.MagicMock()
different_things.side_effect = [1, 2, 3]
assert different_things() == 1 # Pass
assert different_things() == 2 # Pass
assert different_things() == 4 # Fail
- A more realistic example is when simulating file input. In this case, we want to be able to control what `readline` returns each time to `parse_three_lines` input
from unittest import mock
def parse_three_lines(fpin):
line = fpin.readline()
name, value = line.split()
modifier = fpin.readline().strip()
extra = fpin.readline().strip()
return {name: f"{value}/{modifier}+{extra}"}
from io import TextIOBase
def test_parser():
filelike = mock.MagicMock(spec=TextIOBase)
filelike.readline.side_effect = [
"thing important\n",
"a-little\n",
"to-some-people\n"
]
value = parse_three_lines(filelike)
assert value == dict(thing="important/a-little+to-most-people")
Mock Side Effect - Function
------
- Above example is simplified.
- Real network service test code should verify that the results it got were correct to validate that server works correctly.
- This means doing a synthetic request and looking for correct result. The mock object has to emulate that. It has to perform some computation on the inputs.
- Trying to test such code without any computation is difficult.
- The tests tend to be too insensitive or too flaky.
- An insensitive test does not fail in presence of bugs.
- A flaky test is one that sometimes fails, even when the code is correct.
- Below, our code is incorrect and test does not catch it, while flaky test would fail even if it was fixed.
- Mock class fields and methods:
['assert_any_call',
'assert_called',
'assert_called_once',
'assert_called_once_with',
'assert_called_with',
'assert_has_calls',
'assert_not_called',
'attach_mock',
'call_args',
'call_args_list',
'call_count',
'called',
'configure_mock',
'method_calls',
'mock_add_spec',
'mock_calls',
'reset_mock',
'return_value',
'side_effect']
- Read [4]. This is the best write up that I came across.
[1] https://docs.python.org/3/library/unittest.html#
[2] https://docs.python.org/3/library/unittest.mock.html
[3] https://www.toptal.com/python/an-introduction-to-mocking-in-python
[4] https://realpython.com/python-mock-library/
[5] Moshe Zadka https://youtu.be/DJoffYEPttY?list=PL2Uw4_HvXqvYk1Y5P8kryoyd83L_0Uk5K
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
pytest
======
- Tests folder should be same level as project folder
project/
tests/
- Module name (File name) must start with test_*.py
- Method names must start as test_xx()
- Class names must start as class TestSample
Asserting Expected Exceptions
----
def validate_age(age):
if age < 0:
raise ValueError("Age cannot be less than 0")
def test_validate_invalid_age():
with pytest.raises(ValueError): # [1]
validate_age(-1)
- If we don't have [1] context, pytest sees ValueError andd thinks test has
failed.
Markers: Tags for pytests
-----
@pytest.mark.skip(reason="Skipping just for fun")
def test_add():
assert add(1, 2) == 3
@pytest.mark.skipif(sys.version_info > (3, 7), reason="Skipping just for fun")
def test_add():
assert add(1, 2) == 3
# This is most useful marker
@pytest.mark.parameterize("a, b, c",
[(1, 2, 3), ('a', 'b', 'ab')],
ids=["nums", "str", "list"])
def test_add(a, b, c):
assert add(a, b) == c
- 1st arg calls out the parameters
- 2nd arg are values to those parameters
- 3rd arg gives IDs (or names) to those test cases
Fixtures
-------
- Fixture is a function. It is automatically called by pytest when the name of
the argument of another function matches the fixture name. Ex:
@pytest.fixture
def example():
return "foo"
def test_foo(example):
assert example == "foo"
- In this example `example` is called at the moment of execution of test_foo.
- The return value of `example` is passed into `test_foo` as an argument with a name `example`.
- There are many nuances to fixtures (e.g. they have scope, they can use yield instead of return to have some cleanup code, etc).
- Use fixtures when there is canned/repetitive data for every test.
Parameterization
----------------
- It is process of running same test with different values from a prepared set.
Each combination of test and data is counted as new test case.
@pytest.mark.parametrize("number", [1, 2, 3, 0, 42])
def test_foo(number):
assert number > 0
Collection Time
---------------
- pytest has an execution stages called
`collection time`
`test time`
- In this stage, pytest discovers test files and test functions within those
files. pytest also performs "dynamic generation of tests" in this stage.
- Fixtures are also created at this stage, but decorators (such as
@pytest.fixture) are executed at a module import time.
- After collection time finished, Pytest starts the next stage, called ‘test time’.
- In this stage, setup functions are called, fixtures are called, and test
functions (which were discovered/generated at collection time) are executed.
Fixture Parameters
------------------
- Fixtures can have parameters, but they must be iterable (generators, lists,
tuples, sets, etc.). pytest consumes them and converts them to a list.
- There is no lazy evaluation. All iterables are immediately evaluated.
- Each parameter to a fixture is applied to each function using this fixture.
- If a few fixtures are used in one test function, pytest generates a Cartesian
product of parameters of those fixtures.
I did not understand below section
==================================
- To use those parameters, a fixture must consume a special fixture
named ‘request'.
- It provides the special (built-in) fixture with some information on
the function it deals with.
- request also contains request.param which contains one element from
params.
- The fixture called as many times as the number of elements in the
iterable of params argument, and the test function is called with
values of fixtures the same number of times. (basically, the fixture
is called len(iterable) times with each next element of iterable in
the request.param).
Parametrization with pytest_generate_tests
-------------------------------------------
TBD:
Parameters Tips from video [3]
def square(n: int) -> int:
return n * n
@pytest.mark.parameterize(