diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index 1be6cc950fb..b00124faf38 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -699,6 +699,7 @@ peps/pep-0819.rst @emmatyping peps/pep-0820.rst @encukou peps/pep-0821.rst @JelleZijlstra peps/pep-0822.rst @methane +peps/pep-0824.rst @gvanrossum # ... peps/pep-2026.rst @hugovk # ... diff --git a/peps/pep-0824.rst b/peps/pep-0824.rst new file mode 100644 index 00000000000..6d6244b67ce --- /dev/null +++ b/peps/pep-0824.rst @@ -0,0 +1,497 @@ +PEP: 824 +Title: Coalescing operators +Author: Marc Mueller +Sponsor: Guido van Rossum +Discussions-To: Pending +Status: Draft +Type: Standards Track +Created: 31-Jan-2025 +Python-Version: 3.15 + + +Abstract +======== + +This PEP proposes adding two new operators. + +* The "``None``-coalescing" operator ``??`` +* The "Coalescing assignment" operator ``??=`` + +The ``None``-coalescing operator evaluates the left hand side, +checks if it is ``not None`` and, if so, returns the result. If the +value is ``None``, the right hand side is evaluated and returned. + +The coalescing assignment operator will only assign the right hand side +to the left hand side if the left hand side evaluates to ``None``. + +They are roughly equivalent to: + +.. code-block:: python + + # a ?? b + _t if ((_t := a) is not None) else b + + # a ??= b + if a is None: + a = b + +See the `Specification`_ section for more details. + + +Motivation +========== + +First officially proposed ten years ago in (the now deferred) :pep:`505`, +the idea to add ``None``-coalescing operators has been along for some +time now, discussed at length in numerous threads, most recently in +[#discuss_revisit_505]_. This PEP aims to capture the current state of +discussion and proposes a specification for addition to the Python +language. In contrast to :pep:`505`, it will only focus on the two +``None``-coalescing operators. See the `Deferred Ideas`_ section for +more details. + +``None``-coalescing operators are not a new invention. Several other +modern programming languages have so called "``null`` coalescing" +operators, including TypeScript [#ts]_, ECMAScript (a.k.a. JavaScript) +[#js_1]_ [#js_2]_, C# [#csharp]_, Dart [#dart_1]_ [#dart_2]_, +Swift [#swift]_, Kotlin [#kotlin]_, PHP [#php_1]_ [#php_2]_ and more. + +The general idea is to provide a comparison operator which instead of +checking for truthiness, looks for ``None`` values. + +Explicit checks for ``None`` +---------------------------- + +In Python ``None`` is often used to denote the absence of a value. As +such it it common to have to check if a value is ``None`` to do some +action or provide a fallback value. Python provides several options to +do that, each with their own advantages and disadvantages. One such +option is a conditional statement: + +.. code-block:: python + + def func(val: str | None): + if val is not None: + s = val + else: + s = "fallback" + print(f"The value is {s.lower()}") + +While the intent is quite clear, the conditional statement is verbose +and often requires repeating the variable expression. A common +alternative is the conditional expression: + +.. code-block:: python + + def func(val: str | None): + s = val if val is not None else "fallback" + print(f"The value is {s.lower()}") + +This is more concise. It is however still necessary to repeat the +variable expression. If those get more complex, it is often common +that developers choose to introduce a new temporary variable, splitting +the variable from the conditional expression and as such end up +unintentionally introducing additional complexity. + +.. code-block:: python + + def func(obj): + val = obj.var.get_some_value() + s = val if val is not None else "fallback" + print(f"The value is {s.lower()}") + +Furthermore, it is not unusual to see the condition being used in its +"normal" and "inverted" form in the same code base, increasing the +mental load while reading the code. + +.. code-block:: python + + def func(obj): + val = obj.var.get_some_value() + s = "fallback" if val is None else val + print(f"The value is {s.lower()}") + +Another often seen approach is to just forgo the safety of an explicit +conditional expression entirely and use a boolean ``or`` instead. + +.. code-block:: python + + def func(obj): + s = obj.var.get_some_value() or "fallback" + print(f"The value is {s.lower()}") + +This can and often does work fine though it hides potential error cases +which might get unnoticed now. By choosing ``or`` over ``is not None``, +Python will check for truthiness instead of comparing the value to +``None``. It frequently works fine as ``None`` evaluates to ``False``. +Errors can occur though if the variable being checked can be falsy for +values other then ``None``, too. Common ones include ``0``, ``""``, +``()``, ``[]``, ``{}``, ``set()`` or custom objects which overwrite +``__bool__``. + +The "``None``-coalescing" operator can bridge this gab by adding an +alternative to ``or`` which explicitly checks only for ``None`` values. +Adding ``??`` can keep the expression concise while clearly +communicating the intent: + +:: + + def func(obj): + s = obj.var.get_some_value() ?? "fallback" + print(f"The value is {s.lower()}") + + +Overwrite ``None`` values +------------------------- + +Another use case, especially for function argument defaults, is to +overwrite the "old" ``None`` value. + +.. code-block:: python + + def func(val: str | None): + if val is None: + val = "fallback" + print(f"The value is {val.lower()}") + +A new ``??=`` operator is introduced to make these conditional +assignments easier. + +:: + + def func(val: str | None): + val ??= "fallback" + print(f"The value is {val.lower()}") + +It avoids repeating the variable expression which can be especially +helpful for more complex expressions. + +:: + + def other(obj): + if obj.var.some_other_object.another_variable is None: + obj.var.some_other_object.another_variable = "fallback" + + # can be replaced with + obj.var.some_other_object.another_variable ??= "fallback" + + +Specification +============= + +The ``None``-coalescing operator +-------------------------------- + +The ``??`` operator is added. It first evaluates the left hand side. +The result is cached, so that the expression is not evaluated again. +If the value is ``not None``, it is returned. If it is ``None``, the +right hand side expression is evaluated and returned instead. + +.. code-block:: python + + # a ?? b + _t if ((_t := a) is not None) else b + +Precedence +********** + +The precedence of ``??`` will be between ``or`` and conditional +expressions. Parentheses can be added as necessary to modify the +precedence in individual expressions. A few examples of how +implicit parentheses would be placed: + +:: + + # "" or None ?? 2 + ("" or None) ?? 2 + + # "Hello" if None ?? True else 0 + "Hello" if (None ?? True) else 0 + +AST changes +*********** + +A new ``CoalesceOp`` AST node is added. Similarly to ``BoolOp``, it +stores a sequence of subexpressions as ``values``. + +:: + + expr = BoolOp(boolop op, expr* values) + | CoalesceOp(expr* values) + | ... + + +Grammar changes +*************** + +A new ``??`` token is added, as well as a new ``coalesce`` rule. Every +rule which previously referenced the ``disjunction`` rule is updated +to refer to the ``coalesce`` rule instead. + +.. code-block:: PEG + + coalesce: + | disjunction ('??' disjunction)+ + | disjunction + + disjunction: + | conjunction ('or' conjunction)+ + | conjunction + + +The ``Coalescing`` assignment operator +-------------------------------------- + +The ``??=`` operator is added. It performs a conditional assignment. +As such it will first evaluate the left hand side and check that the +value is ``None`` and only then assign the result from the right hand +side. If the first value is ``not None``, the assignment is skipped. + +.. code-block:: python + + # a ??= b + if a is None: + a = b + +AST changes +*********** + +A new ``CoalesceAssign`` AST node is added. Similarly to ``AugAssign``, +it stores a ``target`` and ``value`` expression. + +:: + + stmt = ... + | AugAssign(expr target, operator op, expr value) + ... + | CoalesceAssign(expr target, expr value) + +Grammar changes +*************** + +A new ``??=`` token is added. Additionally, the ``assignment`` rule +is extended to include the coalesce assignment. + +.. code-block:: PEG + + assignment: + | NAME ':' expression ['=' annotated_rhs] + | ('(' single_target ')' + | single_subscript_attribute_target) ':' expression ['=' annotated_rhs] + | (star_targets '=')+ annotated_rhs !'=' [TYPE_COMMENT] + | single_target augassign ~ annotated_rhs + | single_target '??=' ~ annotated_rhs + + +Backwards Compatibility +======================= + +The ``None``-coalescing operators are **opt-in**. Existing programs will +continue to run as is. So far code which used either ``??`` or ``??=`` +raised a ``SyntaxError``. + + +Security Implications +===================== + +There are no new security implications from this proposal. + + +How to Teach This +================= + +In a practical sense it might be helpful to think of the "``None``-coalescing" +operator ``??`` as a special case for the conditional ``or`` +operator, with the caveat that ``??`` checks for ``is not None`` instead +of truthiness. As such it makes sense to include ``??`` when teaching +about the other conditional operators ``and`` and ``or``. + +The "coalescing assignment" ``??=`` operator can be best thought of as +a conditional assignment operator. As it is closely related to ``??`` +explaining these together would make sense. + + +Reference Implementation +======================== + +A reference implementation is available at +https://github.com/cdce8p/cpython/tree/pep824-none-coalescing-operators. +A online demo can be tested at https://pep823-and-pep824-demo.pages.dev/. + + +Deferred Ideas +============== + +``None``-aware access operators +------------------------------- + +:pep:`505` also suggest the addition of the ``None``-aware access +operators ``?.`` and ``?[ ]``. As the ``None``-coalescing operators have +their own use cases, the ``None``-aware access operators were moved +into a separate document, see PEP-823. Both proposals can be adopted +independently of one another. + + +Rejected Ideas +============== + +Add new (soft-) keyword +----------------------- + +Python does have a history of preferring keywords over symbols. For +example, while a lot of languages use ``&&`` and ``||`` as conditional +operators, Python uses ``and`` and ``or`` respectively. As such it was +suggested to use a new (soft-) keyword, e.g. ``otherwise``, instead of +``??``. + +While the keywords ``and`` and ``or`` help avoid ambiguity with the +binary operators ``&`` and ``|``, there is not a corresponding binary +operator for ``??``. Furthermore, both keywords are well established +in spoken and written language and as such immediately obvious to the +reader. Not to mention they are also quite short with just two and three +characters. + +In comparison a new (soft-) keyword would likely not enjoy the same +benefits. There is no established short name for it, so while ideas +like ``otherwise`` could be added, they do not convey an inherit +meaning and therefore do not provide an immediate benefit. In contrast, +the ``??`` operator is well known in other major programming +languages. + +Lastly, using a (soft-) keyword for the "coalescing assignment" operator +poses additional questions and readability concerns. + +:: + + a = otherwise b + +Add ``??`` as a binary operator +------------------------------- + +:pep:`505` originally suggested to add ``??`` as another binary operator. +As such it would have bound more tightly than the proposed +`specification `_. + +Though this would have worked fine, it would have suggested that ``??`` +is similar to other binary operators like ``+`` or ``**``. This is not +the case. While binary operators first evaluate the left **and** right +hand side before performing the operation, for ``??`` only the left hand +side is evaluated if the values is ``not None``. As such the +"``None``-coalescing" operator is much more closely related to the +conditional operators ``or`` and ``and`` which also short-circuit the +expression for truthy and falsy values respectively. + +Furthermore, setting the precedence between ``or`` and conditional +expressions matches other languages which have implemented the operator, +like JS [#js_precedence]_ and C# [#csharp_precedence]_. + +Add ``??=`` as ``AugAssign`` +---------------------------- + +:pep:`505` also suggested to add ``??=`` as an ``AugAssign`` node. + +So far ``AugAssign`` is only used for binary operators, as such including +``??=`` which is a conditional assignment operator would be confusing. +Furthermore, ``AugAssign`` statements always evaluate the left **and** +right hand side, without any short-circuiting. This is a major +difference compared to coalescing assignments. + + +Common objections +================= + +Just use a conditional expression +--------------------------------- + +The ``None``-coalescing operators can be considered syntactic sugar +for existing conditional expressions and statements. As such some +questioned whether they would add anything meaningful to the +language as a whole. + +As shown in the `Motivation`_ section, there are clear benefits to +using the ``None``-coalescing operators. To summarize them again: + +- They help avoid repeating the variable expression or having to + introduce a temporary variable. + +- Clear control flow, no more ``if ... is not None else ...`` and the + inverse ``if ... is None else ...`` in the same code blocks. + +- Avoids the often (mis-) used ``or`` to provide fallback values + just for ``None``. + +- More concise while also being more explicit. + +Proliferation of ``None`` in code bases +--------------------------------------- + +One of the reasons why :pep:`505` stalled was that some expressed their +concern how ``None``-coalescing and ``None``-aware operators will affect +the code written by developers. If it is easier to work with ``None`` +values, this will encourage developers to use them more. They believe that +e.g. returning an optional ``None`` value from a function is usually an +anti-pattern. In an ideal world the use of ``None`` would be limited as +much as possible, for example with early data validation. + +It is certainly true that new language features affect how the language +as a whole develops. Therefore any changes should be considered carefully. +However, just because ``None`` represents an anti-pattern for some, has +not prevented the community as a whole from using it extensively. Rather +the lack of ``None``-coalescing operators has stopped developers from +writing concise expressions and instead often leads to more complex code +or such which can contain subtle errors, see the `Motivation`_ section +for more details. + +``None`` is not special enough +------------------------------ + +Some mentioned that ``None`` is not special enough to warrant dedicated +operators. + +``None``-coalescing operators have been added to a number of other modern +programming languages. Furthermore, adding ``??`` and ``??=`` is +something which was suggested numerous times since :pep:`505` was +first proposed ten years ago. + +In Python ``None`` is frequently used to indicate the absence of +something better or a missing value. As such it is common to look +specifically for ``None`` values, for example, to provide a default +or fallback value. + + +Footnotes +========= + +.. [#discuss_revisit_505] discuss.python.org: Revisiting PEP 505 - None-aware operators + (https://discuss.python.org/t/revisiting-pep-505-none-aware-operators/74568) +.. [#ts] TypeScript: Nullish Coalescing + (https://www.typescriptlang.org/docs/handbook/release-notes/typescript-3-7.html#nullish-coalescing) +.. [#js_1] JavaScript: Nullish coalescing operator (??) + (https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Nullish_coalescing) +.. [#js_2] JavaScript: Nullish coalescing assignment (??=) + (https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Nullish_coalescing_assignment) +.. [#js_precedence] JavaScript: Operator precedence + (https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Operator_precedence#table) +.. [#csharp] C# Reference: ?? and ??= operators + (https://learn.microsoft.com/en-us/dotnet/csharp/language-reference/operators/null-coalescing-operator) +.. [#csharp_precedence] C# Reference: Operator precedence + (https://learn.microsoft.com/en-us/dotnet/csharp/language-reference/operators/#operator-precedence) +.. [#dart_1] Dart: Conditional expressions + (https://dart.dev/language/operators#conditional-expressions) +.. [#dart_2] Dart: Assignment operators + (https://dart.dev/language/operators#assignment-operators) +.. [#swift] Swift: Nil-Coalescing Operator + (https://docs.swift.org/swift-book/documentation/the-swift-programming-language/basicoperators/#Nil-Coalescing-Operator) +.. [#kotlin] Kotlin: Elvis operator + (https://kotlinlang.org/docs/null-safety.html#elvis-operator) +.. [#php_1] PHP: Null Coalesce Operator + (https://wiki.php.net/rfc/isset_ternary) +.. [#php_2] PHP: Null Coalescing Assignment Operator + (https://wiki.php.net/rfc/null_coalesce_equal_operator) + + +Copyright +========= + +This document is placed in the public domain or under the +CC0-1.0-Universal license, whichever is more permissive.