Conversation
eb8680
left a comment
There was a problem hiding this comment.
Thanks for taking a pass at this! It's an interesting case study in its own right, but we should be thinking about how to turn the underlying ideas into library code.
For context, the original motivation for #526 was less CodeAdapt as a standalone example per se and more to explore a modified recipe for handling Template.__apply__ which seemed promising given its broad applicability and the strong empirical results from the CodeAdapt paper:
- Allow the
Templateto reason and call tools as usual, but require it to use a tool that generates and executes code in a context to get a final answer (#549) - Generate and validate Python code for a function with the same signature as the caller
Templateby requiring it to have the same type signature and lexical context and pass any doctests from the docstring of the originalTemplate(#433) - Return the result of applying the generated and validated function to the original inputs to the
Templatecall.
The context-sensitive, persistent-state execute_python_code tool in this PR is interesting as something we could fold into the library as part of a general Template.__apply__ handler, and is closely related to the approach in Recursive Language Models which like CodeAdapt has some strong empirical results.
Something closer to the spirit of the CodeAdapt algorithm than the basic step 2 above would be to generate multiple function bodies that are constrained to pass different subsets of the doctest cases, then tested without further edits on the held-out cases, and select one that successfully extrapolates from its "training" cases to its "test" cases.
There was a problem hiding this comment.
The changes here seem useful on their own, as a solution for #573
| from effectful.ops.semantics import handler | ||
|
|
||
|
|
||
| class DenyImports(RestrictingNodeTransformer): |
There was a problem hiding this comment.
I could be wrong, but I think something like this may already be enforced by RestrictedPython in its safe_globals setting for __import__? Was it not excluding imports by default?
Closes #526