Main verl upgrade by binary-husky · Pull Request #16 · modelscope/AgentJet

binary-husky · 2026-03-19T06:54:00Z

No description provided.

gemini-code-assist · 2026-03-19T06:55:44Z

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly updates the AgentJet project's foundational dependencies, including the verl framework, vllm library, and transformers. These upgrades necessitate corresponding changes across the codebase, such as refactoring worker initialization, introducing custom Ajet-specific extensions for actor configurations, and updating environment variables for enhanced compatibility and performance. The changes aim to leverage the latest features and improvements from the updated libraries, ensuring the project remains current and robust.

Highlights

Core Dependency Upgrades: Upgraded the verl framework to version 0.7.1, vllm to 0.11.0 (or 0.12.0/0.15.0 depending on specific requirements file), and transformers to 4.57.6. The Python version requirement has also been updated to 3.12.
Ajet-Specific Verl Extensions: Introduced custom Ajet-specific worker implementations (AjetActorRolloutRefWorker, AjetAsyncActorRolloutRefWorker, AjetDataParallelPPOActor) and actor configurations (AjetActorConfig, AjetFSDPActorConfig) to extend and override default Verl functionalities, including support for override_ppo_mini_batch_num.
Trainer Refactoring and Simplification: Refactored main_verl.py and trainer_verl.py to streamline worker initialization, remove deprecated ray.init parameters and ray.timeline functionality, and simplify reward function handling. Profiling-related code has also been removed from the training loop.
Configuration and Environment Variable Updates: Updated default configuration files (ajet_default.yaml, verl_default.yaml) with new rollout parameters, detailed optimizer settings, and mappings for new configuration fields. Several environment variables have been added or adjusted for improved VLLM and HCCL compatibility.
Installation and Documentation Alignment: The installation documentation (docs/en/installation.md) and project dependencies (pyproject.toml, requirements_stable_vllm_*.txt) have been updated to reflect the new Python version requirement (3.12) and the specific versions of core libraries.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request introduces a major upgrade to the verl backbone, refactoring core training logic, configuration, and dependencies. Key changes include adopting new Ajet specific workers, overhauling the weight synchronization mechanism to use a CheckpointManager, and introducing a more complex rollout correction algorithm. While these changes seem to align with a newer version of the underlying framework, there are several areas that need attention regarding code robustness, maintainability, and removal of debug artifacts before merging.

gemini-code-assist · 2026-03-19T06:57:37Z

ajet/backbone/trainer_verl.py

+    try:
+        reward_result = reward_fn(data, return_dict=True)
+        reward_tensor = reward_result["reward_tensor"]
+        reward_extra_infos_dict = reward_result.get("reward_extra_info", {})
+    except Exception as e:
+        print(f"Error in reward_fn: {e}")
+        reward_tensor = reward_fn(data)
+        reward_extra_infos_dict = {}
+


The use of a broad except Exception is risky as it can mask unexpected and critical errors during reward computation. This makes debugging difficult and can lead to silent failures. It's better to catch specific exceptions that you expect the legacy reward_fn to raise, or to check the function's signature/return type before calling it.

gemini-code-assist · 2026-03-19T06:57:38Z

ajet/backbone/verl/dp_actor.py

+        # make sure we are in training mode
+        self.actor_module.train()
+
+        from ajet import bp; bp("UPP")


This appears to be a debugging breakpoint. It should be removed before merging into the main branch to avoid accidentally halting execution in production or CI environments.

gemini-code-assist · 2026-03-19T06:57:38Z

ajet/task_rollout/async_llm_bridge.py

+                    for token_id, logprob, decoded_string in zip(token_array, logprob_array, decoded_string_array)  # type: ignore
                ],


Using zip on three separate lists (token_array, logprob_array, decoded_string_array) without a length check is unsafe. If the lists have different lengths for any reason, zip will silently truncate the data to the length of the shortest list, which can lead to subtle bugs and data loss. Please add an assertion to ensure all lists have the same length before iterating.

gemini-code-assist · 2026-03-19T06:57:38Z

ajet/backbone/trainer_verl.py

-        else:
-            raise NotImplementedError
+        assert hasattr(self.async_rollout_manager, "agent_loop_workers")
+        assert len(self.async_rollout_manager.agent_loop_workers) == 1, "Please set `num_workers = 1` in `ajet/default_config/verl/verl_default.yaml`"


The assertion message is brittle as it hardcodes a specific file path. If the configuration file is moved or renamed, this message will become misleading. It's better to provide a more general and helpful message that guides the user to the correct configuration parameter.

Suggested change

assert len(self.async_rollout_manager.agent_loop_workers) == 1, "Please set `num_workers = 1` in `ajet/default_config/verl/verl_default.yaml`"

assert len(self.async_rollout_manager.agent_loop_workers) == 1, "Only a single agent loop worker is supported. Please check your configuration for `ajet.rollout.agent.num_workers` and set it to 1."

gemini-code-assist · 2026-03-19T06:57:38Z

ajet/backbone/verl/dp_actor.py

+                        loss = policy_loss * loss_scale_factor
+                    loss.backward()
+
+                    print(f'*** Current tensor shape, input_ids {input_ids.shape}, response {response_mask.shape}, loss {loss}, loss_scale_factor {loss_scale_factor}')


This print statement seems to be for debugging purposes. It should be removed to avoid cluttering the logs in production.

gemini-code-assist · 2026-03-19T06:57:38Z

ajet/task_runner/base_runner.py

+        # # run gc in a thread-safe way
+        # if gc_lock.acquire(blocking=False):
+        #     try:
+        #         gc.collect()
+        #     finally:
+        #         gc_lock.release()


This block of code for garbage collection is commented out without any explanation. If it's no longer necessary, it should be removed to improve code clarity. If it's disabled temporarily, a comment explaining the reason (e.g., performance issues, handled elsewhere) would be very helpful for future maintenance.

gemini-code-assist · 2026-03-19T06:57:38Z

pyproject.toml

 requires-python = ">=3.10,<3.13"
 dependencies = [
-    "agentscope==1.0.8",
+    "agentscope==1.0.7",


Downgrading agentscope from 1.0.8 to 1.0.7 is an unusual change. It would be beneficial to add a comment explaining why this downgrade is necessary (e.g., to avoid a specific regression or bug in the newer version). This context is valuable for anyone managing dependencies in the future.

binary-husky and others added 12 commits March 17, 2026 01:16

set up

a18048f

stage success merge

edabb22

support qwen-3.5

67e8092

patch for higher version vllm

ad08055

stable vllm and verl 0.8.0

7909969

patch docker builder

8e2a954

upgrade python to 3.12

b36bf73

patch green verl versions

1828d58

supress tool parser warnings

0c96884

align benchmarking hyper-parameters

01d3fa3

remove outdated argument

8f59278

add a loss scaler

60acbdf

gemini-code-assist bot reviewed Mar 19, 2026

View reviewed changes

binary-husky force-pushed the main-verl-upgrade branch from d0dd93f to 60acbdf Compare March 21, 2026 15:42

binary-husky added 2 commits March 21, 2026 23:55

fix broken arguments

563d570

rename source code

3a92c53

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Main verl upgrade#16

Main verl upgrade#16
binary-husky wants to merge 14 commits intomainfrom
main-verl-upgrade

binary-husky commented Mar 19, 2026

Uh oh!

gemini-code-assist bot commented Mar 19, 2026

Uh oh!

gemini-code-assist bot left a comment

Uh oh!

gemini-code-assist bot Mar 19, 2026

Uh oh!

gemini-code-assist bot Mar 19, 2026

Uh oh!

gemini-code-assist bot Mar 19, 2026

Uh oh!

gemini-code-assist bot Mar 19, 2026

Uh oh!

gemini-code-assist bot Mar 19, 2026

Uh oh!

gemini-code-assist bot Mar 19, 2026

Uh oh!

gemini-code-assist bot Mar 19, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

		for token_id, logprob, decoded_string in zip(token_array, logprob_array, decoded_string_array) # type: ignore
		],

	assert len(self.async_rollout_manager.agent_loop_workers) == 1, "Please set `num_workers = 1` in `ajet/default_config/verl/verl_default.yaml`"
	assert len(self.async_rollout_manager.agent_loop_workers) == 1, "Only a single agent loop worker is supported. Please check your configuration for `ajet.rollout.agent.num_workers` and set it to 1."

Conversation

binary-husky commented Mar 19, 2026

Uh oh!

gemini-code-assist bot commented Mar 19, 2026

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist bot Mar 19, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Mar 19, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Mar 19, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Mar 19, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Mar 19, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Mar 19, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Mar 19, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant