👀

[julien-blanchon]

Hey,

I'm interested by what's happening here ! The current stack for RL training for diffusion is awfull (flow-grpo forks ...)
I've been following https://github.com/chenyingshu/mm_grpo recently but the project seems to be on hold.

Your codebase seems really good already. Do you have any ETA ? Are you working alone on this ?

[Jayce-Ping]

Hi! Glad to have attention here!

I'm currently working alone on this. I don't actually have a specific ETA, since I'm thinking through the logic while coding. However, the current version could be published with a bit more testing and guidance. I'm making small progress every day.

The project you mentioned seems more ambitious, especially regarding support for multi-model generative frameworks based on verl and vLLM. For this codebase, I plan to focus exclusively on flow-matching and diffusion models first. This scope is much more achievable for a single developer.

Btw, this code base is just a draft with millions of debugging and commits history. I'll compress history after or create a clean new repo for publishment.😊

[julien-blanchon]

I think it's a better idea to only focus on flow-matching and diffusion, research is still evolving too fast to make a generalist framework.

Have you already run hours long training with the currrent codebase (like pick_score on flux or qwen), have been able to reproduce flow-grpo results ?

[Jayce-Ping]

Reproducing results is not a problem. I have tested Flux.1-dev, Qwen-Image, and FLUX2, and they are all fine-tuning on PickScore as expected. However, for Z-Image-Turbo, the improvement is very small. I suspect it has already been tuned on PickScore, which I confirmed in their tech report.

Currently, I'm focusing on model sharding—specifically using FSDP Full Shard or FSDP2—to make full-parameter tuning more achievable.

If you want to reproduce some result of specific model, I can help you to design a best config, since I've already very familiar with these staff. Glad to help!😊 (If you use the config in the repo directly, maybe some bugs are not really fixed.)

I'll release a overall guidance maybe after several days. There are still some bugs to fix and I want to make everything perfect.

[MengHao666]

Great work!
could u consider to support the following algorithms:

1. GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimization
2. GARDO: Reinforcing Diffusion Models without Reward Hacking
3. Reinforcing Diffusion Models by Direct Group Preference Optimization

Aslo， why not support tensorboard for logging? As i only use logging in pure offline scence.

[Jayce-Ping]

Great work! could u consider to support the following algorithms:

1. GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimization
2. GARDO: Reinforcing Diffusion Models without Reward Hacking
3. Reinforcing Diffusion Models by Direct Group Preference Optimization

Aslo， why not support tensorboard for logging? As i only use logging in pure offline scence.

Hi, thanks for pointing these out!

The algorithms: DPO, GARDO and GDPO are all on the TODO list. I will add support for multi-reward first, then GDPO and GARDO, finally to DPO.

As far as I know, wandb and swanlab both support offline logging. As for tensorboard logger, I will try to support it later.

[MengHao666]

Great work! could u consider to support the following algorithms:

1. GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimization
2. GARDO: Reinforcing Diffusion Models without Reward Hacking
3. Reinforcing Diffusion Models by Direct Group Preference Optimization

Aslo， why not support tensorboard for logging? As i only use logging in pure offline scence.

Hi, thanks for pointing these out!

The algorithms: DPO, GARDO and GDPO are all on the TODO list. I will add support for multi-reward first, then GDPO and GARDO, finally to DPO.

As far as I know, wandb and swanlab both support offline logging. As for tensorboard logger, I will try to support it later.

Bravo! Hope we could use flow-factory as easy as Lllam-Factory! Your project is very meaningful if tested by many community users!

I mean sometimes users may want to visulize all logs in their own local computers to protect the data privacy of private enterprise users. I found wandbandswanlab` could only support visualize only after uploading to website, which is good for research usage, but not for private enterprise users.

[julien-blanchon]

I'm also planning to help testing and contributing on this library. I really see the value @Jayce-Ping is putting here for the community

[MengHao666]

I'm also planning to help testing and contributing on this library. I really see the value @Jayce-Ping is putting here for the community

Make Flow-Factory great as Lllama-Factory!

[MengHao666]

Also, you can refer to here to modify original grpo-guard implementations. The author said, (Note: diving dt can make training unstable on some flagship models.). However, it might need more experiments to validate it.

For combination of flow-cps and GRPO-Guard, you could actually refer to yifan123/flow_grpo#192 (comment)

Also, I stronly suggest to make the reward computation more flexible then flow-grpo. For example, UnifiedReward-Think now is the most powerful open-source reward models that I have ever tested, it could get better results then any other open source models. Please check Pref-GRPO) to see its good performance. But it is a pair-wise preference reward models, it only could be only computed after whole epoch sampling process. And oher point-wise rewards could be just computed after very small mini-batch.

And for very long view, the following method which do RL combined with distillation could be added:

1. Decoupled-DMD: The Acceleration Magic Behind Z-Image
2. DMDR: Fusing DMD with Reinforcement Learning

@Jayce-Ping

[Jayce-Ping]

Hi! Thanks for your attention and enthusiasm.

The author said, (Note: dividing dt can make training unstable on some flagship models.). However, it might need more experiments to validate it.

From my experience, I think grpo-guard is overclaimed. So, I'm still settling for original GRPO loss, which works quite well. Maybe I'm wrong. We need more experiments to validate, as you said.

For combination of flow-cps and GRPO-Guard, you could actually refer to yifan123/flow_grpo#192 (comment)

Thanks for pointing this out! I'll look into this.

Also, I strongly suggest to make the reward computation more flexible than flow-grpo. For example, UnifiedReward-Think now is the most powerful open-source reward model that I have ever tested, it could get better results than any other open source models. Please check Pref-GRPO to see its good performance. But it is a pair-wise preference reward model, it can only be computed after whole epoch sampling process. And other point-wise rewards could be computed after very small mini-batch.

I know Pref-GRPO and also ran some experiments. My impression was that it's way too slow! Since it requires pair-wise computation among the whole group, resulting in quadratic time-complexity. But anyway, thanks for pointing this out and I will try to make it compatible. As far as I know, there are some other algorithms like Branch-GRPO/Tree-GRPO, I don't know how to add them yet. From my experience, Flow-GRPO-Fast/Mix-GRPO's idea provides the best performance and training speed, that's why I prioritize it.

And for very long view, the following method which do RL combined with distillation could be added:
Decoupled-DMD: The Acceleration Magic Behind Z-Image
DMDR: Fusing DMD with Reinforcement Learning

Great! I can try to add them.

After all, you guys are warmly welcomed to bring up any issues or create PRs for contribution.🤗 Thanks again.❤️

PS: About naming Flow-Factory, it is indeed named after Llama-factory, which is a very good repo. Also, ff is quite easy to type and makes the training launching easier, Haha.😃

[MengHao666]

I agree with the idea that grpo-guard is overclaimed. I really can not see big imrovements after grpo-guard. and it would also introduce many hper-paramters which are very hard to tune.

We could see Branch-GRPO and Tree-GRPO did not draw much attentions from the community users. I can not see the big value to add them in your repo even they may have their own academic values.

As we could see the best open source t2i models (Qwen-Image 2512 or Z-image-Turbo) could achieve very high photorealism and text-rendering ability. The old reward models actually do not work for these sota models. Those who play with SD-3.5 and pickscore all day to argue their RL methods or reward models would be forgotted quickly.

As for my experience, once works for T2I models like old Qwen-Image or Flux-dev.1, the RL methods or reward models could be proved to be useful. Once works for T2I models like Qwen-Image-2512 or Z-image, the RL methods or reward models could be proved to be a breakthrough for industry usage .

The Pref-GRPO and Unified-Reward-Think works for old Qwen-Image!

by the way, UnifiedReward-Think-qwen3vl-2b is enough for experiments validation.

[Jayce-Ping]

Hi, I've added TensorBoard support. However, I'm running experiments on a remote server, and the connection to my local PC is quite slow, making it difficult to render anything locally. I don’t have much experience with TensorBoard, so please feel free to test its functionality and provide feedback. PRs are also welcome!

[MengHao666]

Thanks for quick support. Also, would u consider add nft training for qwen-image or qwen-image-2512?

[Jayce-Ping]

Also, would u consider add nft training for qwen-image or qwen-image-2512?

Hi! All the combinations of listed models and algorithms are supported now. The existing examples are all tested and parameter fine-tuned. For nft + qwen-image/qwen-image-2512, I will update an config file based on my inituition and verify later.

[Nijikadesu]

Nice work!

Recently I was trying to write RL training code for Wan2.1-I2V based on codebase of flow_grpo, and that’s when I started thinking about a more general RL framework for training generative models. So it really excites me to see someone in the community share the same idea and put it into practice. Hope this project progresses smoothly. 🫡

Would be happy to contribute to this exciting project if there’s an opportunity. Is there any roadmap or future plans that I can start with? 😊