Inlined comprehensions

Since PEP 709, list, set, and dict comprehensions are compiled into the enclosing compilation unit instead of creating a nested function and calling it. Generator expressions are not inlined; they still become their own code object.

The resulting bytecode lives in the enclosing unit, but the comprehension still has its own locals: iteration variables must not leak into, or overwrite, names in the enclosing scope. The symbol table models that as a nested lexical scope; codegen then emits the comprehension inlined into its containing compilation unit.

Which comprehensions are inlined

symtable_handle_comprehension() in Python/symtable.c inlines a comprehension when it is not a generator expression and the current block cannot see class scope (!ste_can_see_class_scope). Annotation scopes that can see a class keep the historical nested-function compilation so class-local names are not treated as comprehension locals.

The outermost iterator expression is always evaluated in the enclosing scope. The rest of the comprehension (targets, if clauses, inner generators, and the element/value expressions) is visited in the comprehension's own block.

Symbol table

An inlined comprehension gets an InlinedComprehensionBlock entry (pycore_symtable.h). That entry is a child of the enclosing block, with ste_parent pointing at the enclosing PySTEntryObject. It is a lexical subscope, not a compilation unit: there is no separate code object, co_consts entry, or compiler scope for it.

Uses and bindings inside the comprehension are recorded on that child table. Because those loads are in the enclosing compilation unit, a FREE use is also marked USE on the parent table. For def inner(): return [x for y in ()], both inner.lookup("x") and the inlined child's lookup are FREE|USE.

Analysis

analyze_block() records this block's declarations, analyzes children, then classifies uses. finalize_inlined_comprehension() copies USE from an inlined child onto the parent before that second pass, and drops inlined-only frees so analyze_cells() does not promote those names to cells.

A name that is FREE in the comprehension and bound in the parent is dropped from the parent's free set unless:

  • a real nested unit (function, lambda, or genexp) still needs it as a cell, or
  • a sibling nested scope already marked it free.

That keeps iteration variables as fast locals when they are only used by nested inlined comprehensions. A nested lambda that captures the name still forces a cell on the binding comprehension.

Class-closure names (__class__ and friends) loaded from an inlined comprehension do not require a class cell unless a nested function, lambda, or genexp captures them. Compile treats the inlined loads as implicit globals. is_free_in_any_child() walks through inlined children and only counts FREE on non-inlined descendants.

Compiler

Codegen stays in the enclosing compiler unit. Around the inlined region, _PyCompile_EnterInlinedComprehensionScope() / _PyCompile_ExitInlinedComprehensionScope() in Python/compile.c swap c->u->u_ste so name lookup uses the comprehension's symbol table. The saved u_ste is restored on both success and error.

_PyCompile_ResolveNameop() calls compiler_resolve_inlined_free(), which walks ste_parent while the current table is inlined and the name is FREE or missing (scope 0). Missing names include loads synthesized by codegen, such as the implicit receiver for zero-argument super(). The walk stops at a class: nested scopes do not see class locals. Class-closure names that would otherwise be free through a class become GLOBAL_IMPLICIT.

Isolating iteration variables

codegen_push_inlined_comprehension_locals() in Python/codegen.c isolates names bound in the comprehension:

  • LOAD_FAST_AND_CLEAR saves the enclosing value (possibly NULL) and clears the slot.
  • MAKE_CELL runs if the name is a cell for this comprehension.
  • In module and class units the name is added to u_fasthidden so assemble can set CO_FAST_HIDDEN.

A SETUP_FINALLY / COMPILE_FBLOCK_INLINED_COMPREHENSION handler restores those slots if the comprehension raises, so an enclosing except or finally sees the original values.

Runtime

An inlined comprehension cell can share a localsplus name with an enclosing free variable (for example [lambda: x for x in x] inside a nested function). FrameLocalsProxy keys, values, items, and len keep the first slot of each name so they agree with getitem.

Source