Skip to content

Comments

Add simple codeadapt example#596

Open
jfeser wants to merge 1 commit intomasterfrom
jf-codeadapt
Open

Add simple codeadapt example#596
jfeser wants to merge 1 commit intomasterfrom
jf-codeadapt

Conversation

@jfeser
Copy link
Contributor

@jfeser jfeser commented Feb 24, 2026

Closes #526

Copy link
Contributor

@eb8680 eb8680 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for taking a pass at this! It's an interesting case study in its own right, but we should be thinking about how to turn the underlying ideas into library code.

For context, the original motivation for #526 was less CodeAdapt as a standalone example per se and more to explore a modified recipe for handling Template.__apply__ which seemed promising given its broad applicability and the strong empirical results from the CodeAdapt paper:

  1. Allow the Template to reason and call tools as usual, but require it to use a tool that generates and executes code in a context to get a final answer (#549)
  2. Generate and validate Python code for a function with the same signature as the caller Template by requiring it to have the same type signature and lexical context and pass any doctests from the docstring of the original Template (#433)
  3. Return the result of applying the generated and validated function to the original inputs to the Template call.

The context-sensitive, persistent-state execute_python_code tool in this PR is interesting as something we could fold into the library as part of a general Template.__apply__ handler, and is closely related to the approach in Recursive Language Models which like CodeAdapt has some strong empirical results.

Something closer to the spirit of the CodeAdapt algorithm than the basic step 2 above would be to generate multiple function bodies that are constrained to pass different subsets of the doctest cases, then tested without further edits on the held-out cases, and select one that successfully extrapolates from its "training" cases to its "test" cases.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The changes here seem useful on their own, as a solution for #573

from effectful.ops.semantics import handler


class DenyImports(RestrictingNodeTransformer):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I could be wrong, but I think something like this may already be enforced by RestrictedPython in its safe_globals setting for __import__? Was it not excluding imports by default?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Recreate CodeAdapt paper - handler for templates that completes query by generating code

2 participants