fix: aligns GRPOConfig with the upstream trl && update docs

[Tohrusky]

PR type

• Bug Fix
• New Feature
• Document Updates
• More Models or Datasets Support

PR information

This PR aligns the GRPOConfig logic in ms-swift with the upstream trl implementation to ensure consistent behavior regarding generation_batch_size.

See:

ms-swift/swift/rlhf_trainers/arguments.py

Line 107 in b52bf59

self.generation_batch_size = self.per_device_train_batch_size * num_processes * self.steps_per_generation

And trl:
https://github.com/huggingface/trl/blob/main/trl/trainer/grpo_config.py#L843

Docs Updates

Fixed a error in docs/source/Instruction/GRPO/GetStarted/GRPO.md.

Correction: When steps_per_generation is set, the total generation_batch_size should be calculated as: generation_batch_size = per_device_train_batch_size * steps_per_generation * num_processes

Previously, the documentation was misleading regarding how steps_per_generation's default value impacts the generation_batch_size.

Additional Question

The Megatron implementation seems to behave differently with swift:

https://github.com/modelscope/ms-swift/blob/b52bf591fbc1e054908fec064f794536307c3c89/swift/megatron/trainers/grpo_trainer.py#L1071C21-L1071C41

https://swift.readthedocs.io/zh-cn/v3.12/Megatron-SWIFT/GRPO.html#id1

The doc here is a bit confusing regarding steps_per_generation.
Is this difference in logic intentional due to Megatron?
If so, it might be helpful to update the docs to clearly distinguish between the standard behavior and the Megatron behavior to avoid confusion.

Thanks! ❤

[gemini-code-assist]

Summary of Changes

Hello @Tohrusky, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request standardizes the configuration logic for generation_batch_size and steps_per_generation within the GRPOConfig to match the behavior of the upstream trl library. This ensures consistent and predictable operation of the GRPO trainer. Concurrently, the associated documentation has been thoroughly updated to accurately reflect these changes and provide clearer guidance on parameter interactions, thereby enhancing user understanding and preventing potential misconfigurations.

Highlights

• GRPOConfig Alignment: The logic for handling generation_batch_size and steps_per_generation within GRPOConfig has been updated in swift/rlhf_trainers/arguments.py to align with the upstream trl library's implementation, ensuring consistent behavior.
• Documentation Updates: Comprehensive updates have been made to both Chinese and English documentation files (docs/source/Instruction/Command-line-parameters.md, docs/source/Instruction/GRPO/GetStarted/GRPO.md, docs/source_en/Instruction/Command-line-parameters.md, docs/source_en/Instruction/GRPO/GetStarted/GRPO.md) to accurately reflect the corrected calculation of generation_batch_size and clarify related GRPO parameters.
• Parameter Configuration Clarity: The documentation now provides clearer explanations for conditions leading to off-policy training and improved guidance on configuring mini-batch sizes in GRPO, including refined examples.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog

• docs/source/Instruction/Command-line-parameters.md
  • Reordered the descriptions of steps_per_generation and generation_batch_size for better logical flow.
  • Corrected the default calculation formula for generation_batch_size.
  • Simplified the divisibility requirement for num_generations.
• docs/source/Instruction/GRPO/GetStarted/GRPO.md
  • Updated the formula for total sampling batch size when steps_per_generation is active.
  • Clarified the default calculation for generation_batch_size when steps_per_generation defaults to gradient_accumulation_steps.
  • Added '或者' (or) to the condition for off-policy training.
  • Streamlined and improved the explanation and example for configuring mini-batch size.
• docs/source_en/Instruction/Command-line-parameters.md
  • Reordered parameter descriptions and updated the default generation_batch_size calculation.
  • Simplified the divisibility requirement for num_generations.
• docs/source_en/Instruction/GRPO/GetStarted/GRPO.md
  • Corrected the total sampling batch size formula for steps_per_generation.
  • Clarified the default generation_batch_size calculation.
  • Added 'or' to the off-policy training condition.
  • Refined the mini-batch configuration explanation and example.
• swift/rlhf_trainers/arguments.py
  • Implemented new conditional logic in __post_init__ to align generation_batch_size and steps_per_generation initialization with trl's grpo_config.py.
  • Ensured that if generation_batch_size is set, steps_per_generation is derived from it, and vice-versa.
  • Added a ValueError to prevent simultaneous configuration of both generation_batch_size and steps_per_generation.
  • Removed a redundant divisibility check for generation_batch_size as it's now implicitly handled by the new logic.

Activity

• The author, Tohrusky, submitted this pull request to address a bug related to GRPOConfig alignment and to update documentation.
• The author raised a specific question in the PR description regarding potential differences in steps_per_generation logic for Megatron implementation, suggesting a need for further clarification in the documentation.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request aligns the GRPOConfig logic with the upstream trl implementation and updates the corresponding documentation. The changes in swift/rlhf_trainers/arguments.py correctly handle the initialization of generation_batch_size and steps_per_generation, making the logic more robust and explicit. The documentation updates in both Chinese and English versions accurately reflect these changes, correcting previous inaccuracies and improving clarity for users. The code is well-structured and the documentation is now more precise. Overall, this is a solid improvement.