DeFlowCritic: Dense Latent Reward Alignment for Text-to-Image Flow Matching Models

Abstract

Online reinforcement learning (RL) has emerged as a powerful paradigm for aligning text-to-image flow matching models with complex user intent. However, existing methods typically rely on sparse rewards computed only from final decoded images. This delays credit assignment across the denoising trajectory and makes training computationally expensive. Moreover, simply combining multiple rewards does not reliably produce a Pareto trade-off, where gains in compositional accuracy often degrade aesthetic quality. In this work, we propose DeFlowCritic, an efficient online RL framework based on Dense Latent Reward Alignment. Unlike prior work, our method provides dense supervision on intermediate latents, enabling early-stage guidance for the denoising trajectory. To achieve this, we introduce an internal critic that evaluates noisy latents using features extracted directly from the diffusion model, a representation inherently better suited for latent-space reward estimation than standard image encoders. We construct a high-quality reward-modeling dataset with continuous scores for critic training, capturing compositional correctness on high-quality reference images. The resulting dense reward enables efficient online RL training and inference-time scaling. Experiments show that our method achieves a superior balance between prompt faithfulness and visual quality on GenEval2 and T2I-CompBench++ benchmarks. The approach also generalizes across text-to-image flow matching models, including SD3.5-Medium, SD3.5-Large, and FLUX.1-dev.

How It Works

Training

Train a latent critic, then use it as a dense reward

DeFlowCritic overview: (a) the critic scores a noisy latent against a positive prompt and negatives; (b) GRPO with dense latent rewards

Train DeFlowCritic

Frozen SD3.5-M with small LoRA adapters and query tokens learns to score a noisy latent against the right prompt and graded wrong ones.

GRPO with dense rewards

The frozen critic rewards every denoising step; early steps are scored after a short ODE rollout.

Critic training data

Graded wrong prompts

Each reference image comes with its prompt (1.00) and wrong prompts graded by how much they change.

Results

Critic

Latent flow-matching critic beats VLMs on noisy images

To reward intermediate steps, the critic has to judge images that are still noisy. A VLM such as Qwen-3-VL only sees pixels, so the noisy latent must first be decoded into a noisy image. Zero-shot, its ranking accuracy falls from 58% to 5% as noise grows; even fine-tuned on the same data, it drops to 67%. DeFlowCritic scores the latent directly with the generator's own features, which were pretrained on the same noisy latents that RL visits, and stays above 91% at every noise level.

Method	Reward input	Fine-tuned	t = 465	602	707	780
Qwen-3-VL (zero-shot)	Decoded image	✗	58.41	22.66	12.73	5.23
Qwen-3-VL (fine-tuned, noisy images)	Decoded image	✓	92.16	85.07	77.64	67.22
DeFlowCritic	Latent	✓	97.08	96.67	95.83	91.03

Ranking accuracy on held-out images: how often the right prompt scores above all wrong ones. Higher t = noisier.

GRPO

Comparable composition, higher quality, 10× faster

GRPO on SD3.5-M. Composition close to Flow-GRPO, the best HPSv3 on both benchmarks, at a tenth of the training time. (Trained on GenEval-1 prompts, so GenEval2 and T2I-CompBench++ are out-of-domain.)

Model	GenEval2		T2I-CompBench++		Training time ↓
Model	Comp. ↑	HPSv3 ↑	Comp. ↑	HPSv3 ↑	Training time ↓
SD3.5-M	18.10	6.66	52.37	7.76	–
Flow-GRPO (GenEval)	23.40	7.13	59.44	9.26	1×
fast Flow-GRPO (GenEval)	22.70	4.45	58.64	5.71	0.1×
fast Flow-GRPO (GenEval + Aesth.)	12.50	6.89	55.24	8.72	0.12×
fast Flow-GRPO (DeFlowCritic)	21.44	8.07	59.37	10.34	0.1×

Latent reward models

Stronger than preference-trained latent rewards

Other latent reward models learn from huge sets of human preference pairs: 0.58M for SLRM and 0.8M for DINA-LRM. DeFlowCritic is trained on only 5K reference images, about a hundred times fewer examples, yet the policy it trains is about 22 points higher on GenEval-1 composition, with image quality close to theirs.

Method (SD3.5-M)	Reward data	GenEval-1	HPSv2.1	HPSv3
SD3.5-M	–	55.34	27.15	8.29
SLRM	0.58M preference pairs	68.93	32.01	9.89
DINA-LRM	0.8M preference pairs	68.15	32.41	9.69
DeFlowCritic	5K images	90.65	30.71	9.62

Inference-time scaling

Better test-time selection for less compute

Because DeFlowCritic scores noisy latents, it can judge candidates part-way through denoising: we start 8 candidates, score them early, and finish only the best. CLIPScore needs a clean image, so all 8 must be fully generated before it can pick. DeFlowCritic picks better, with fewer denoiser calls. (Inference-time scaling on GenEval2 prompts.)

Method	Denoiser calls	Sec / image	GenEval2	HPSv3
SD3.5-M	80	1.4	18.10	6.66
+ CLIPScore, best of 8	640	7.3	21.95	6.58
+ DeFlowCritic, best of 8	444	5.1	24.09	7.32

BibTeX

@inproceedings{khan2026deflowcritic,
  title     = {Dense Latent Reward Alignment for Text-to-Image Flow Matching Models},
  author    = {Khan, Zeeshan and Yu, Xin and Chen, Shizhe and Schmid, Cordelia},
  booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
  year      = {2026}
}

Acknowledgements

This work was granted access to the HPC resources of IDRIS under the allocation AD011014688R2 made by GENCI. It was funded in part by the French government under management of Agence Nationale de la Recherche as part of the “France 2030” program, reference ANR-23-IACL-0008 (PR[AI]RIE-PSAI projet), and Paris Île-de-France Région in the frame of the DIM AI4IDF. Cordelia Schmid would like to acknowledge the support by the Körber European Science Prize.

Copyright

The documents contained in these directories are included by the contributing authors as a means to ensure timely dissemination of scholarly and technical work on a non-commercial basis. Copyright and all rights therein are maintained by the authors or by other copyright holders, notwithstanding that they have offered their works here electronically. It is understood that all persons copying this information will adhere to the terms and constraints invoked by each author's copyright.