[
{
"path": "README.md",
"content": "# Awesome Evaluation of Visual Generation\n\n\n*This repository collects methods for evaluating visual generation.*\n\n\n\n## Overview\n\n### What You'll Find Here\n\nWithin this repository, we collect works that aim to answer some critical questions in the field of evaluating visual generation, such as:\n\n- **Model Evaluation**: How does one determine the quality of a specific image or video generation model?\n- **Sample/Content Evaluation**: What methods can be used to evaluate the quality of a particular generated image or video?\n- **User Control Consistency Evaluation**: How to tell how well the generated images and videos align with the user controls or inputs?\n\n### Updates\n\nThis repository is updated periodically. If you have suggestions for additional resources, updates on methodologies, or fixes for expiring links, please feel free to do any of the following:\n- raise an [Issue](https://github.com/ziqihuangg/Awesome-Evaluation-of-Visual-Generation/issues),\n- nominate awesome related works with [Pull Requests](https://github.com/ziqihuangg/Awesome-Evaluation-of-Visual-Generation/pulls),\n- We are also contactable via email (`ZIQI002 at e dot ntu dot edu dot sg`).\n\n### Table of Contents\n- [1. Evaluation Metrics of Generative Models](#1.)\n - [1.1. Evaluation Metrics of Image Generation](#1.1.)\n - [1.2. Evaluation Metrics of Video Generation](#1.2.)\n - [1.3. Evaluation Metrics for Latent Representation](#1.3.)\n- [2. Evaluation Metrics of Condition Consistency](#2.)\n - [2.1 Evaluation Metrics of Multi-Modal Condition Consistency](#2.1.)\n - [2.2. Evaluation Metrics of Image Similarity](#2.2.)\n- [3. Evaluation Systems of Generative Models](#3.)\n - [3.1. Evaluation of Unconditional Image Generation](#3.1.)\n - [3.2. Evaluation of Text-to-Image Generation](#3.2.)\n - [3.3. Evaluation of Text-Based Image Editing](#3.3.)\n - [3.4. Evaluation of Neural Style Transfer](#3.4.)\n - [3.5. Evaluation of Video Generation](#3.5.)\n - [3.6. Evaluation of Text-to-Motion Generation](#3.6.)\n - [3.7. Evaluation of Model Trustworthiness](#3.7.)\n - [3.8. Evaluation of Entity Relation](#3.8.)\n - [3.9. Agentic Evaluation](#3.9.)\n- [4. Improving Visual Generation with Evaluation / Feedback / Reward](#4.)\n- [5. Quality Assessment for AIGC](#5.)\n- [6. Study and Rethinking](#6.)\n- [7. Other Useful Resources](#7.)\n\n\n## 1. Evaluation Metrics of Generative Models\n\n### 1.1. Evaluation Metrics of Image Generation\n\n\n| Metric | Paper | Code |\n| -------- | -------- | ------- |\n| Inception Score (IS) | [Improved Techniques for Training GANs](https://arxiv.org/abs/1606.03498) (NeurIPS 2016) | |\n| Fréchet Inception Distance (FID) | [GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium](https://arxiv.org/abs/1706.08500) (NeurIPS 2017) | [](https://github.com/bioinf-jku/TTUR) [](https://github.com/mseitzer/pytorch-fid) |\n| Kernel Inception Distance (KID) | [Demystifying MMD GANs](https://arxiv.org/abs/1801.01401) (ICLR 2018) | [](https://github.com/toshas/torch-fidelity) [](https://github.com/NVlabs/stylegan2-ada-pytorch/blob/main/metrics/kernel_inception_distance.py) \n| CLIP-FID | [The Role of ImageNet Classes in Fréchet Inception Distance](https://arxiv.org/abs/2203.06026) (ICLR 2023) | [](https://github.com/kynkaat/role-of-imagenet-classes-in-fid) [](https://github.com/GaParmar/clean-fid?tab=readme-ov-file#computing-clip-fid) |\n| Precision-and-Recall |[Assessing Generative Models via Precision and Recall](https://arxiv.org/abs/1806.00035) (2018-05-31, NeurIPS 2018)
[Improved Precision and Recall Metric for Assessing Generative Models](https://arxiv.org/abs/1904.06991) (NeurIPS 2019) | [](https://github.com/msmsajjadi/precision-recall-distributions) [](https://github.com/kynkaat/improved-precision-and-recall-metric) |\n| Renyi Kernel Entropy (RKE) | [An Information-Theoretic Evaluation of Generative Models in Learning Multi-modal Distributions](https://openreview.net/forum?id=PdZhf6PiAb) (NeurIPS 2023) | [](https://github.com/mjalali/renyi-kernel-entropy) |\n| CLIP Maximum Mean Discrepancy (CMMD) | [Rethinking FID: Towards a Better Evaluation Metric for Image Generation](https://arxiv.org/abs/2401.09603) (CVPR 2024) | [](https://github.com/google-research/google-research/tree/master/cmmd) |\n| Fréchet Wavelet Distance (FWD) | [Fréchet Wavelet Distance: A Domain-Agnostic Metric For Image Generation](https://openreview.net/pdf?id=QinkNNKZ3b) (ICLR 2025) | [](https://github.com/BonnBytes/PyTorch-FWD) |\n\n\n+ [Towards a Scalable Reference-Free Evaluation of Generative Models](https://arxiv.org/abs/2407.02961) (2024-07-03)\n\n+ [FaceScore: Benchmarking and Enhancing Face Quality in Human Generation](https://arxiv.org/abs/2406.17100) (2024-06-24)\n >Note: Face Score introduced\n\n+ [Global-Local Image Perceptual Score (GLIPS): Evaluating Photorealistic Quality of AI-Generated Images](https://arxiv.org/abs/2405.09426) (2024-05-15)\n\n+ [Unifying and extending Precision Recall metrics for assessing generative models](https://arxiv.org/abs/2405.01611) (2024-05-02)\n\n+ [Enhancing Plausibility Evaluation for Generated Designs with Denoising Autoencoder](https://arxiv.org/abs/2403.05352) (2024-03-08) \n >Note: Fréchet Denoised Distance introduced\n\n+ Virtual Classifier Error (VCE) from [Virtual Classifier: A Reversed Approach for Robust Image Evaluation](https://openreview.net/forum?id=IE6FbueT47) (2024-03-04)\n\n+ [An Interpretable Evaluation of Entropy-based Novelty of Generative Models](https://arxiv.org/abs/2402.17287) (2024-02-27)\n\n+ Semantic Shift Rate from [Discovering Universal Semantic Triggers for Text-to-Image Synthesis](https://arxiv.org/abs/2402.07562) (2024-02-12)\n\n+ [Optimizing Prompts Using In-Context Few-Shot Learning for Text-to-Image Generative Models](https://ieeexplore.ieee.org/document/10378642) (2024-01-01)\n >Note: Quality Loss introduced\n\n+ [Attribute Based Interpretable Evaluation Metrics for Generative Models](https://arxiv.org/abs/2310.17261) (2023-10-26) \n\n+ [On quantifying and improving realism of images generated with diffusion](https://arxiv.org/abs/2309.14756) (2023-09-26)\n >Note: Image Realism Score introduced\n \n+ [Probabilistic Precision and Recall Towards Reliable Evaluation of Generative Models](https://arxiv.org/abs/2309.01590) (2023-09-04) \n[](https://github.com/kdst-team/Probablistic_precision_recall)\n >Note: P-precision and P-recall introduced\n\n+ [Learning to Evaluate the Artness of AI-generated Images](https://arxiv.org/abs/2305.04923) (2023-05-08)\n >Note: ArtScore, metric for images resembling authentic artworks by artists\n\n+ [Training-Free Location-Aware Text-to-Image Synthesis](https://arxiv.org/abs/2304.13427) (2023-04-26) \n > Note: New evaluation metric for control capability of location aware generation task\n\n+ [Feature Likelihood Divergence: Evaluating the Generalization of Generative Models Using Samples](https://arxiv.org/abs/2302.04440) (2023-02-09)\n[](https://github.com/marcojira/fld)\n\n+ [LGSQE: Lightweight Generated Sample Quality Evaluatoin](https://arxiv.org/abs/2211.04590) (2022-11-08)\n\n+ [SSD: Towards Better Text-Image Consistency Metric in Text-to-Image Generation](https://arxiv.org/abs/2210.15235) (2022-10-27)\n > Note: Semantic Similarity Distance introduced\n\n+ [Layout-Bridging Text-to-Image Synthesis](https://arxiv.org/abs/2208.06162) (2022-08-12)\n > Note: Layout Quality Score (LQS), new metric for evaluating the generated layout\n\n+ [Rarity Score: A New Metric to Evaluate the Uncommonness of Synthesized Images](https://arxiv.org/abs/2206.08549) (2022-06-17) \n[](https://github.com/hichoe95/Rarity-Score)\n\n+ [Mutual Information Divergence: A Unified Metric for Multimodal Generative Models](https://arxiv.org/abs/2205.13445) (2022-05-25)\n[](https://github.com/naver-ai/mid.metric)\n >Note: evaluates text to image and utilizes vision language models (VLM)\n\n\n+ [TREND: Truncated Generalized Normal Density Estimation of Inception Embeddings for GAN Evaluation](https://arxiv.org/abs/2104.14767) (2021-04-30, ECCV 2022)\n\n+ CFID from [Conditional Frechet Inception Distance](https://arxiv.org/abs/2103.11521) (2021-03-21)\n[](https://github.com/Michael-Soloveitchik/CFID/)\n[](https://michael-soloveitchik.github.io/CFID/)\n\n+ [On Self-Supervised Image Representations for GAN Evaluation](https://openreview.net/forum?id=NeRdBeTionN) (2021-01-12)\n[](https://github.com/stanis-morozov/self-supervised-gan-eval)\n > Note: SwAV, self-supervised image representation model\n\n+ [Random Network Distillation as a Diversity Metric for Both Image and Text Generation](https://arxiv.org/abs/2010.06715) (2020-10-13)\n >Note: RND metric introduced\n\n+ [The Vendi Score: A Diversity Evaluation Metric for Machine Learning](https://arxiv.org/abs/2210.02410) (2022-10-05) \n[](https://github.com/vertaix/Vendi-Score)\n\n+ CIS from [Evaluation Metrics for Conditional Image Generation](https://arxiv.org/abs/2004.12361) (2020-04-26)\n\n+ [Text-To-Image Synthesis Method Evaluation Based On Visual Patterns](https://arxiv.org/abs/1911.00077) (2020-04-09) \n\n+ [Cscore: A Novel No-Reference Evaluation Metric for Generated Images](https://dl.acm.org/doi/abs/10.1145/3373509.3373546) (2020-03-25) \n\n\n+ SceneFID from [Object-Centric Image Generation from Layouts](https://arxiv.org/abs/2003.07449) (2020-03-16)\n\n+ [Reliable Fidelity and Diversity Metrics for Generative Models](https://arxiv.org/abs/2002.09797) (2020-02-23, ICML 2020) \n[](https://github.com/clovaai/generative-evaluation-prdc) \n\n+ [Effectively Unbiased FID and Inception Score and where to find them](https://arxiv.org/abs/1911.07023) (2019-11-16, CVPR 2020) \n[](https://github.com/mchong6/FID_IS_infinity)\n\n+ [On the Evaluation of Conditional GANs](https://arxiv.org/abs/1907.08175) (2019-07-11)\n >Note:Fréchet Joint Distance (FJD), which is able to assess image quality, conditional consistency, and intra-conditioning diversity within a single metric.\n\n+ [Quality Evaluation of GANs Using Cross Local Intrinsic Dimensionality](https://arxiv.org/abs/1905.00643) (2019-05-02)\n > CrossLID, assesses the local intrinsic dimensionality \n\n+ [A domain agnostic measure for monitoring and evaluating GANs](https://arxiv.org/abs/1811.05512) (2018-11-13) \n\n+ [Learning to Generate Images with Perceptual Similarity Metrics](https://arxiv.org/abs/1511.06409) (2015-11-19)\n > Multiscale structural-similarity score introduced\n\n+ [A No-Reference Image Blur Metric Based on the Cumulative Probability of Blur Detection (CPBD)](https://ieeexplore.ieee.org/document/5739529) (2011-03-28) \n\n\n\n### 1.2. Evaluation Metrics of Video Generation\n\n\n| Metric | Paper | Code |\n| -------- | -------- | ------- |\n| FID-vid | [GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium](https://arxiv.org/abs/1706.08500) (NeurIPS 2017) | |\n| Fréchet Video Distance (FVD) | [Towards Accurate Generative Models of Video: A New Metric & Challenges](https://arxiv.org/abs/1812.01717) (arXiv 2018)
[FVD: A new Metric for Video Generation](https://openreview.net/forum?id=rylgEULtdN) (2019-05-04) (Note: ICLR 2019 Workshop DeepGenStruct Program Chairs)| [](https://github.com/songweige/TATS/blob/main/tats/fvd/fvd.py) |\n\n\n### 1.3. Evaluation Metrics for Latent Representation\n\n+ Linear Separability & Perceptual Path Length (PPL) from [A Style-Based Generator Architecture for Generative Adversarial Networks](https://arxiv.org/abs/1812.04948) (2020-01-09)\n[](https://github.com/NVlabs/stylegan?tab=readme-ov-file)\n[](https://github.com/NVlabs/ffhq-dataset)\n\n\n\n## 2. Evaluation Metrics of Condition Consistency\n\n### 2.1 Evaluation Metrics of Multi-Modal Condition Consistency\n\n\n| Metric | Condition | Pipeline | Code | References | \n| -------- | -------- | ------- | -------- | -------- | \n| CLIP Score (`a.k.a.` CLIPSIM) | Text | cosine similarity between the CLIP image and text embeddings | [](https://github.com/openai/CLIP) [PyTorch Lightning](https://lightning.ai/docs/torchmetrics/stable/multimodal/clip_score.html) | [CLIP Paper](https://arxiv.org/abs/2103.00020) (ICML 2021). Metrics first used in [CLIPScore Paper](https://arxiv.org/abs/2104.08718) (arXiv 2021) and [GODIVA Paper](https://arxiv.org/abs/2104.14806) (arXiv 2021) applies it in video evaluation. |\n| Mask Accuracy | Segmentation Mask | predict the segmentatio mask, and compute pixel-wise accuracy against the ground-truth segmentation mask | any segmentation method for your setting |\n| DINO Similarity | Image of a Subject (human / object *etc*) | cosine similarity between the DINO embeddings of the generated image and the condition image | [](https://github.com/facebookresearch/dino) | [DINO paper](https://arxiv.org/abs/2104.14294). Metric is proposed in [DreamBooth](https://arxiv.org/abs/2208.12242).\n\n\n\n\n\n+ NexusScore, NaturalScore and GmeScore from [OpenS2V-Nexus: A Detailed Benchmark and Million-Scale Dataset for Subject-to-Video Generation](https://arxiv.org/abs/2505.20292) (2025-06-03)\n >Note: NexusScore - Identity Consistency - image retrieval + cosine similarity; NaturalScore - Identity Naturalness - prompting gpt4o; GmeScore - Text - cosine similarity between the GME image and text embeddings.\n\n+ FaceSim-Cur from [Identity-Preserving Text-to-Video Generation by Frequency Decomposition](https://arxiv.org/abs/2411.17440) (2024-11-26)\n >Note: NFaceSim-Cur - Face image of human - cosine similarity between the curricularface embeddings of the generated face and the input face.\n\n+ Manipulation Direction (MD) from [Manipulation Direction: Evaluating Text-Guided Image Manipulation Based on Similarity between Changes in Image and Text Modalities](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10675000/) (2023-11-20)\n\n+ [Semantic Similarity Distance: Towards better text-image consistency metric in text-to-image generation](https://www-sciencedirect-com.remotexs.ntu.edu.sg/science/article/pii/S0031320323005812?via%3Dihub) (2022-12-02)\n\n+ [On the Evaluation of Conditional GANs](https://arxiv.org/abs/1907.08175) (2019-07-11)\n >Note: Fréchet Joint Distance (FJD), which is able to assess image quality, conditional consistency, and intra-conditioning diversity within a single metric.\n\n+ [Classification Accuracy Score for Conditional Generative Models](https://arxiv.org/abs/1905.10887) (2019-05-26)\n > Note: New metric Classification Accuracy Score (CAS)\n\n\n+ Visual-Semantic (VS) Similarity from [Photographic Text-to-Image Synthesis with a Hierarchically-nested Adversarial Network](https://arxiv.org/abs/1802.09178v2) (2018-12-26)\n[](https://github.com/ypxie/HDGan)\n[](https://alexhex7.github.io/2018/05/30/Photographic%20Text-to-Image%20Synthesis%20with%20a%20Hierarchically-nested%20Adversarial%20Network/)\n\n\n+ [Semantically Invariant Text-to-Image Generation](https://arxiv.org/abs/1809.10274) (2018-09-06)\n[](https://github.com/sxs4337/MMVR)\n > Note: They evaluate image-text similarity via image captioning\n\n+ [Inferring Semantic Layout for Hierarchical Text-to-Image Synthesis](https://arxiv.org/abs/1801.05091v2) (2018-01-16) \n > Note: An object detector based metric is proposed.\n\n\n\n\n### 2.2. Evaluation Metrics of Image Similarity\n\n| Metrics | Paper | Code |\n| -------- | -------- | ------- |\n| Learned Perceptual Image Patch Similarity (LPIPS) | [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://arxiv.org/abs/1801.03924) (2018-01-11) (CVPR 2018) | [](https://github.com/richzhang/PerceptualSimilarity) [](https://richzhang.github.io/PerceptualSimilarity/) |\n| Structural Similarity Index (SSIM) | [Image quality assessment: from error visibility to structural similarity](https://ieeexplore.ieee.org/document/1284395) (TIP 2004) | [](https://github.com/open-mmlab/mmagic/blob/main/tests/test_evaluation/test_metrics/test_ssim.py) [](https://github.com/Po-Hsun-Su/pytorch-ssim) |\n| Peak Signal-to-Noise Ratio (PSNR) | - | [](https://github.com/open-mmlab/mmagic/blob/main/tests/test_evaluation/test_metrics/test_psnr.py) |\n| Multi-Scale Structural Similarity Index (MS-SSIM) | [Multiscale structural similarity for image quality assessment](https://ieeexplore.ieee.org/document/1292216) (SSC 2004) | [PyTorch-Metrics](https://lightning.ai/docs/torchmetrics/stable/image/multi_scale_structural_similarity.html#:~:text=Compute%20MultiScaleSSIM%2C%20Multi%2Dscale%20Structural,details%20at%20different%20resolution%20scores.&text=a%20method%20to%20reduce%20metric%20score%20over%20labels.) |\n| Feature Similarity Index (FSIM) | [FSIM: A Feature Similarity Index for Image Quality Assessment](https://ieeexplore.ieee.org/document/5705575) (TIP 2011) | [](https://github.com/mikhailiuk/pytorch-fsim)\n\n\n\nThe community has also been using [DINO](https://arxiv.org/abs/2104.14294) or [CLIP](https://arxiv.org/abs/2103.00020) features to measure the semantic similarity of two images / frames.\n\n\nThere are also recent works on new methods to measure visual similarity (more will be added):\n\n+ [DreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic Data](https://arxiv.org/abs/2306.09344) (2023-06-15) \n [](https://github.com/ssundaram21/dreamsim)\n [](https://dreamsim-nights.github.io)\n \n\n## 3. Evaluation Systems of Generative Models\n\n\n### 3.1. Evaluation of Unconditional Image Generation\n\n+ [AesBench: An Expert Benchmark for Multimodal Large Language Models on Image Aesthetics Perception](https://arxiv.org/abs/2401.08276) (2024-01-16) \n\n+ [A Lightweight Generalizable Evaluation and Enhancement Framework for Generative Models and Generated Samples](https://ieeexplore.ieee.org/document/10495634) (2024-04-16) \n\n+ [Anomaly Score: Evaluating Generative Models and Individual Generated Images based on Complexity and Vulnerability](https://arxiv.org/abs/2312.10634) (2023-12-17, CVPR 2024) \n\n+ [Using Skew to Assess the Quality of GAN-generated Image Features](https://arxiv.org/abs/2310.20636) (2023-10-31) \n > Note: Skew Inception Distance introduced\n\n+ [StudioGAN: A Taxonomy and Benchmark of GANs for Image Synthesis](https://arxiv.org/abs/2206.09479) (2022-06-19)\n[](https://github.com/POSTECH-CVLab/PyTorch-StudioGAN) [](https://huggingface.co/Mingguksky/PyTorch-StudioGAN/tree/main)\n\n\n\n+ [HYPE: A Benchmark for Human eYe Perceptual Evaluation of Generative Models](https://arxiv.org/abs/1904.01121) (2019-04-01) \n[](https://stanfordhci.github.io/gen-eval/)\n\n+ [An Improved Evaluation Framework for Generative Adversarial Networks](https://arxiv.org/abs/1803.07474) (2018-03-20) \n > Note: Class-Aware Frechet Distance introduced\n\n\n\n### 3.2. Evaluation of Text-to-Image Generation\n+ [GenExam: A Multidisciplinary Text-to-Image Exam](https://arxiv.org/abs/2509.14232) (2025-09-18)\n [](https://github.com/OpenGVLab/GenExam)\n\n+ [What Makes a Scene ? Scene Graph-based Evaluation and Feedback for Controllable Generation](https://arxiv.org/abs/2411.15435) (2024-05-26)\n\n+ [Commonsense-T2I Challenge: Can Text-to-Image Generation Models Understand Commonsense?](https://arxiv.org/abs/2406.07546) (2024-08-12)\n [](https://zeyofu.github.io/CommonsenseT2I/)\n\n+ [WISE: A World Knowledge-Informed Semantic Evaluation for Text-to-Image Generation](https://arxiv.org/abs/2503.07265) (2025-05-27)\n [](https://github.com/PKU-YuanGroup/WISE)\n\n+ [Why Settle for One? Text-to-ImageSet Generation and Evaluation](https://arxiv.org/abs/2506.23275) (2025-06-29)\n\n+ [LMM4LMM: Benchmarking and Evaluating Large-multimodal Image Generation with LMMs](https://arxiv.org/abs/2504.08358) (2025-04-11)\n [](https://github.com/IntMeGroup/LMM4LMM)\n\n+ [Robust and Discriminative Speaker Embedding via Intra-Class Distance Variance Regularization](https://www.isca-archive.org/interspeech_2018/le18_interspeech.html) (2018-09)\n >Note: IntraClass Average Distance(ICAD) - Diversity \n\n+ [REAL: Realism Evaluation of Text-to-Image Generation Models for Effective Data Augmentation](https://arxiv.org/abs/2502.10663). (2025-02-15)\n\n+ [Evaluation Agent: Efficient and Promptable Evaluation Framework for Visual Generative Models](https://arxiv.org/abs/2412.09645) (2024-12-16)\n [](https://github.com/Vchitect/Evaluation-Agent) \n [](https://vchitect.github.io/Evaluation-Agent-project/)\n >Note: focus on efficient and dynamic evaluation. \n\n\n+ [ABHINAW: A method for Automatic Evaluation of Typography within AI-Generated Images](https://arxiv.org/abs/2409.11874) (2024-09-18)\n[](https://github.com/Abhinaw3906/ABHINAW-MATRIX)\n\n+ [Finding the Subjective Truth: Collecting 2 Million Votes for Comprehensive Gen-AI Model Evaluation](https://arxiv.org/abs/2409.11904) (2024-09-18)\n\n+ [Beyond Aesthetics: Cultural Competence in Text-to-Image Models](https://arxiv.org/abs/2407.06863) (2024-07-09)\n > Note: CUBE benchmark introduced\n\n+ [MJ-Bench: Is Your Multimodal Reward Model Really a Good Judge for Text-to-Image Generation?](https://arxiv.org/abs/2407.04842) (2024-07-05) \n > Note: MJ-Bench introduced\n\n+ [MIGC++: Advanced Multi-Instance Generation Controller for Image Synthesis](https://arxiv.org/abs/2407.02329) (2024-07-02) \n[](https://github.com/limuloo/MIGC)\n[](https://migcproject.github.io/)\n > Note: Benchmark COCO-MIG and Multimodal-MIG introduced\n\n\n+ [Analyzing Quality, Bias, and Performance in Text-to-Image Generative Models](https://arxiv.org/abs/2407.00138) (2024-06-28)\n\n\n+ [EvalAlign: Evaluating Text-to-Image Models through Precision Alignment of Multimodal Large Models with Supervised Fine-Tuning to Human Annotations](https://arxiv.org/abs/2406.16562) (2024-06-24)\n[](https://github.com/SAIS-FUXI/EvalAlign)\n[](https://huggingface.co/Fudan-FUXI/evalalign-v1.0-13b)\n\n\n+ [DreamBench++: A Human-Aligned Benchmark for Personalized Image Generation](https://arxiv.org/abs/2406.16855) (2024-06-24)\n[](https://github.com/yuangpeng/dreambench_plus)\n[](https://dreambenchplus.github.io/)\n\n\n+ [Six-CD: Benchmarking Concept Removals for Benign Text-to-image Diffusion Models](https://arxiv.org/pdf/2406.14855) (2024-06-21) \n[](https://github.com/Artanisax/Six-CD)\n\n+ [Evaluating Numerical Reasoning in Text-to-Image Models](https://arxiv.org/abs/2406.14774) (2024-06-20) \n > Note: GeckoNum introduced\n\n+ [Holistic Evaluation for Interleaved Text-and-Image Generation](https://arxiv.org/abs/2406.14643) (2024-06-20) \n > Note: InterleavedBench and InterleavedEval metric introduced\n\n+ [GenAI-Bench: Evaluating and Improving Compositional Text-to-Visual Generation](https://arxiv.org/abs/2406.13743) (2024-06-19)\n\n+ [Decomposed evaluations of geographic disparities in text-to-image models](https://arxiv.org/abs/2406.11988) (2024-06-17)\n[](https://ai.meta.com/research/publications/decomposed-evaluations-of-geographic-disparities-in-text-to-image-models/)\n > Note: new metric Decomposed Indicators of Disparities introduced\n\n+ [PhyBench: A Physical Commonsense Benchmark for Evaluating Text-to-Image Models](https://arxiv.org/abs/2406.11802) (2024-06-17) \n[](https://github.com/OpenGVLab/PhyBench)\n > Note: PhyBench introduced\n\n+ [Make It Count: Text-to-Image Generation with an Accurate Number of Objects](https://arxiv.org/abs/2406.10210) (2024-06-14)\n[](https://github.com/Litalby1/make-it-count)\n[](https://make-it-count-paper.github.io/)\n\n+ [Commonsense-T2I Challenge: Can Text-to-Image Generation Models Understand Commonsense?](https://arxiv.org/abs/2406.07546) (2024-06-11)\n[](https://github.com/zeyofu/Commonsense-T2I)\n[](https://zeyofu.github.io/CommonsenseT2I/)\n[](https://huggingface.co/datasets/CommonsenseT2I/CommonsensenT2I)\n > Note: Commonsense-T2I, benchmark for real-life commonsense reasoning capabilities of T2I models\n\n+ [Unified Text-to-Image Generation and Retrieval](https://arxiv.org/abs/2406.05814) (2024-06-09)\n > Note: TIGeR-Bench, benchmark for evaluation of unified text-to-image generation and retrieval.\n\n+ [PQPP: A Joint Benchmark for Text-to-Image Prompt and Query Performance Prediction](https://arxiv.org/abs/2406.04746) (2024-06-07)\n[](https://github.com/Eduard6421/PQPP)\n\n+ [GenAI Arena: An Open Evaluation Platform for Generative Models](https://arxiv.org/abs/2406.04485) (2024-06-06) \n[](https://github.com/TIGER-AI-Lab/VideoGenHub?tab=readme-ov-file)\n\n+ [A-Bench: Are LMMs Masters at Evaluating AI-generated Images?](https://arxiv.org/abs/2406.03070) (2024-06-05) \n [](https://github.com/Q-Future/A-Bench) [](https://a-bench-sjtu.github.io/) [](https://huggingface.co/datasets/q-future/A-Bench)\n\n+ Multidimensional Preference Score from [Learning Multi-dimensional Human Preference for Text-to-Image Generation](https://arxiv.org/abs/2405.14705) (2024-05-23)\n\n+ [Evolving Storytelling: Benchmarks and Methods for New Character Customization with Diffusion Models](https://arxiv.org/abs/2405.11852) (2024-05-20) \n >Note: NewEpisode benchmark introduced\n\n+ [Training-free Subject-Enhanced Attention Guidance for Compositional Text-to-image Generation](https://arxiv.org/abs/2405.06948) (2024-05-11) \n >Note: GroundingScore metric introduced\n\n+ [TheaterGen: Character Management with LLM for Consistent Multi-turn Image Generation](https://arxiv.org/abs/2404.18919) (2024-04-29)\n[](https://github.com/donahowe/Theatergen)\n[](https://howe140.github.io/theatergen.io/)\n >Note: consistent score r introduced\n\n+ [Exposing Text-Image Inconsistency Using Diffusion Models](https://arxiv.org/abs/2404.18033) (2024-04-28) \n\n+ [Revisiting Text-to-Image Evaluation with Gecko: On Metrics, Prompts, and Human Ratings](https://arxiv.org/abs/2404.16820) (2024-04-25) \n\n+ [Multimodal Large Language Model is a Human-Aligned Annotator for Text-to-Image Generation](https://arxiv.org/abs/2404.15100) (2024-04-23) \n\n+ [Infusion: Preventing Customized Text-to-Image Diffusion from Overfitting](https://arxiv.org/abs/2404.14007) (2024-04-22)\n >Note: Latent Fisher divergence and Wasserstein metric introduced\n\n+ [TAVGBench: Benchmarking Text to Audible-Video Generation](https://arxiv.org/abs/2404.14381) (2024-04-22) \n [](https://github.com/OpenNLPLab/TAVGBench)\n\n+ [Object-Attribute Binding in Text-to-Image Generation: Evaluation and Control](https://arxiv.org/abs/2404.13766) (2024-04-21) \n\n+ [Magic Clothing: Controllable Garment-Driven Image Synthesis](https://arxiv.org/abs/2404.09512) (2024-04-15) \n[](https://github.com/ShineChen1024/MagicClothing)\n[](https://huggingface.co/ShineChen1024/MagicClothing)\n > Note: new metric Matched-Points-LPIPS introduced\n\n+ [GenAI-Bench: A Holistic Benchmark for Compositional Text-to-Visual Generation](https://openreview.net/forum?id=hJm7qnW3ym) (2024-04-09)\n > Note: GenAI-Bench was introduced in a previous paper 'Evaluating Text-to-Visual Generation with Image-to-Text Generation'\n\n+ Detect-and-Compare from [Identity Decoupling for Multi-Subject Personalization of Text-to-Image Models](https://arxiv.org/abs/2404.04243) (2024-04-05) \n[](https://github.com/agwmon/MuDI)\n[](https://mudi-t2i.github.io/)\n\n+ [Enhancing Text-to-Image Model Evaluation: SVCS and UCICM](https://ieeexplore.ieee.org/abstract/document/10480770) (2024-04-02)\n > Note: Evaluation metrics: Semantic Visual Consistency Score and User-Centric Image Coherence Metric \n\n+ [Evaluating Text-to-Visual Generation with Image-to-Text Generation](https://arxiv.org/abs/2404.01291) (2024-04-01) \n [](https://github.com/linzhiqiu/t2v_metrics)\n [](https://linzhiqiu.github.io/papers/vqascore)\n\n+ [Measuring Style Similarity in Diffusion Models](https://arxiv.org/abs/2404.01292) (2024-04-01) \n [](https://github.com/learn2phoenix/CSD)\n\n+ [AAPMT: AGI Assessment Through Prompt and Metric Transformer](https://arxiv.org/abs/2403.19101) (2024-03-28)\n[](https://github.com/huskydoge/CS3324-Digital-Image-Processing/tree/main/Assignment1)\n\n\n+ [FlashEval: Towards Fast and Accurate Evaluation of Text-to-image Diffusion Generative Models](https://arxiv.org/abs/2403.16379) (2024-03-25)\n\n+ [Refining Text-to-Image Generation: Towards Accurate Training-Free Glyph-Enhanced Image Generation](https://arxiv.org/abs/2403.16422) (2024-03-25) \n > Note: LenCom-Eval introduced\n\n+ [Exploring GPT-4 Vision for Text-to-Image Synthesis Evaluation](https://openreview.net/forum?id=xmQoodG82a) (2024-03-20)\n\n+ [DialogGen: Multi-modal Interactive Dialogue System for Multi-turn Text-to-Image Generation](https://arxiv.org/abs/2403.08857) (2024-03-13) \n[](https://github.com/Centaurusalpha/DialogGen)\n > Note: DialogBen introduced\n\n+ [Evaluating Text-to-Image Generative Models: An Empirical Study on Human Image Synthesis](https://arxiv.org/abs/2403.05125) (2024-03-08) \n\n+ [An Information-Theoretic Evaluation of Generative Models in Learning Multi-modal Distributions](https://openreview.net/forum?id=PdZhf6PiAb) (2024-02-13) \n [](https://github.com/mjalali/renyi-kernel-entropy)\n\n+ [MIGC: Multi-Instance Generation Controller for Text-to-Image Synthesis](https://arxiv.org/abs/2402.05408) (2024-02-08) \n[](https://github.com/limuloo/MIGC)\n[](https://migcproject.github.io/)\n > Note: COCO-MIG benchmark introduced\n\n+ [CAS: A Probability-Based Approach for Universal Condition Alignment Score](https://openreview.net/forum?id=E78OaH2s3f) (2024-01-16) \n [](https://github.com/unified-metric/unified_metric) [](https://unified-metric.github.io/)\n > Note: Condition alignment of text-to-image, {instruction, image}-to-image, edge-/scribble-to-image, and text-to-audio\n\n+ [EmoGen: Emotional Image Content Generation with Text-to-Image Diffusion Models](https://arxiv.org/abs/2401.04608) (2024-01-09) \n[](https://github.com/JingyuanYY/EmoGen)\n >Note: emotion accuracy, semantic clarity and semantic diversity are not core contributions of this paper\n\n+ [VIEScore: Towards Explainable Metrics for Conditional Image Synthesis Evaluation](https://arxiv.org/abs/2312.14867) (2023-12-22) \n [](https://github.com/TIGER-AI-Lab/VIEScore) [](https://tiger-ai-lab.github.io/VIEScore/)\n\n+ [PIA: Your Personalized Image Animator via Plug-and-Play Modules in Text-to-Image Models](https://arxiv.org/abs/2312.13964) (2023-12-21)\n[](https://github.com/open-mmlab/PIA) [](https://pi-animator.github.io/)\n > Note: AnimateBench, benchmark for comparisons in the field of personalized image animation\n\n+ [Stellar: Systematic Evaluation of Human-Centric Personalized Text-to-Image Methods](https://arxiv.org/abs/2312.06116) (2023-12-11) \n [](https://github.com/stellar-gen-ai/stellar-metrics)\n [](https://stellar-gen-ai.github.io/)\n\n\n+ [A Contrastive Compositional Benchmark for Text-to-Image Synthesis: A Study with Unified Text-to-Image Fidelity Metrics](https://arxiv.org/abs/2312.02338) (2023-12-04) \n [](https://github.com/zhuxiangru/Winoground-T2I)\n\n+ [The Challenges of Image Generation Models in Generating Multi-Component Images](https://arxiv.org/abs/2311.13620) (2023-11-22) \n\n\n+ [SelfEval: Leveraging the discriminative nature of generative models for evaluation](https://arxiv.org/abs/2311.10708) (2023-11-17)\n\n\n+ [GPT-4V(ision) as a Generalist Evaluator for Vision-Language Tasks](https://arxiv.org/abs/2311.01361) (2023-11-02)\n\n+ [Davidsonian Scene Graph: Improving Reliability in Fine-grained Evaluation for Text-to-Image Generation](https://arxiv.org/abs/2310.18235) (2023-10-27, ICLR 2024) \n [](https://github.com/j-min/DSG)\n [](https://google.github.io/dsg/)\n\n+ [DEsignBench: Exploring and Benchmarking DALL-E 3 for Imagining Visual Design](https://arxiv.org/abs/2310.15144) (2023-10-23) \n [](https://design-bench.github.io)\n \n\n+ [GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment](https://arxiv.org/abs/2310.11513) (2023-10-17) \n [](https://github.com/djghosh13/geneval) \n\n+ [Hypernymy Understanding Evaluation of Text-to-Image Models via WordNet Hierarchy](https://arxiv.org/abs/2310.09247) (2023-10-13) \n [](https://github.com/yandex-research/text-to-img-hypernymy) \n\n+ [SingleInsert: Inserting New Concepts from a Single Image into Text-to-Image Models for Flexible Editing](https://arxiv.org/abs/2310.08094) (2023-10-12) \n [](https://github.com/JarrentWu1031/SingleInsert) [](https://jarrentwu1031.github.io/SingleInsert-web/)\n > Note: New Metric: Editing Success Rate \n\n+ [ImagenHub: Standardizing the evaluation of conditional image generation models](https://arxiv.org/abs/2310.01596) (2023-10-02) \n [](https://github.com/TIGER-AI-Lab/ImagenHub) [](https://tiger-ai-lab.github.io/ImagenHub/) \n [](https://huggingface.co/ImagenHub)\n\n+ [Navigating Text-To-Image Customization: From LyCORIS Fine-Tuning to Model Evaluation](https://arxiv.org/abs/2309.14859) (2023-09-26, ICLR 2024) \n [](https://github.com/KohakuBlueleaf/LyCORIS)\n\n+ Concept Score from [Text-to-Image Generation for Abstract Concepts](https://paperswithcode.com/paper/text-to-image-generation-for-abstract) (2023-09-26) \n\n+ [OpenLEAF: Open-Domain Interleaved Image-Text Generation and Evaluation](https://openreview.net/forum?id=SeiL55hCnu) (2023-09-23) \n [](https://huggingface.co/ImagenHub) [GenAI-Arena](https://huggingface.co/papers/2310.07749)\n > Note: Evaluates task of image and text generation\n \n+ [Progressive Text-to-Image Diffusion with Soft Latent Direction](https://arxiv.org/abs/2309.09466) (2023-09-18)\n[](https://github.com/babahui/Progressive-Text-to-Image)\n >Note: Benchmark for text-to-image generation tasks\n\n\n+ [AltDiffusion: A Multilingual Text-to-Image Diffusion Model](https://arxiv.org/abs/2308.09991) (2023-08-19, AAAI 2024)\n[](https://github.com/superhero-7/AltDiffusion)\n >Note: Benchmark with focus on multilingual generation aspect\n\n\n\n\n\n\n\n+ LEICA from [Likelihood-Based Text-to-Image Evaluation with Patch-Level Perceptual and Semantic Credit Assignment](https://arxiv.org/abs/2308.08525) (2023-08-16)\n \n+ [Let's ViCE! Mimicking Human Cognitive Behavior in Image Generation Evaluation](https://arxiv.org/abs/2307.09416) (2023-07-18) \n\n+ [T2I-CompBench: A Comprehensive Benchmark for Open-world Compositional Text-to-image Generation](https://arxiv.org/abs/2307.06350) (2023-07-12) \n [](https://github.com/Karine-Huang/T2I-CompBench)\n [](https://karine-h.github.io/T2I-CompBench/)\n\n+ [TIAM -- A Metric for Evaluating Alignment in Text-to-Image Generation](https://arxiv.org/abs/2307.05134) (2023-07-11, WACV 2024) \n [](https://github.com/grimalPaul/TIAM)\n\n+ [Divide, Evaluate, and Refine: Evaluating and Improving Text-to-Image Alignment with Iterative VQA Feedback](https://arxiv.org/abs/2307.04749) (2023-07-10, NeurIPS 2023) \n [](https://github.com/1jsingh/Divide-Evaluate-and-Refine) [](https://1jsingh.github.io/divide-evaluate-and-refine)\n\n+ [Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesis](https://arxiv.org/abs/2306.09341) (2023-06-15) \n [](https://github.com/tgxs002/HPSv2)\n\n+ [ConceptBed: Evaluating Concept Learning Abilities of Text-to-Image Diffusion Models](https://arxiv.org/abs/2306.04695) (2023-06-07, AAAI 2024) \n [](https://github.com/ConceptBed/evaluations) [](https://conceptbed.github.io/) [](https://huggingface.co/spaces/mpatel57/ConceptBed)\n\n+ [Visual Programming for Text-to-Image Generation and Evaluation](https://arxiv.org/abs/2305.15328) (2023-05-24, NeurIPS 2023) \n [](https://github.com/aszala/VPEval) [](https://vp-t2i.github.io/)\n\n+ [LLMScore: Unveiling the Power of Large Language Models in Text-to-Image Synthesis Evaluation](https://arxiv.org/abs/2305.11116) (2023-05-18, NeurIPS 2023) \n [](https://github.com/YujieLu10/LLMScore)\n\n+ [X-IQE: eXplainable Image Quality Evaluation for Text-to-Image Generation with Visual Large Language Models](https://arxiv.org/abs/2305.10843) (2023-05-18) \n [](https://github.com/Schuture/Benchmarking-Awesome-Diffusion-Models)\n\n+ [What You See is What You Read? Improving Text-Image Alignment Evaluation](https://arxiv.org/abs/2305.10400) (2023-05-17, NeurIPS 2023) \n [](https://github.com/yonatanbitton/wysiwyr) [](https://wysiwyr-itm.github.io/) [](https://huggingface.co/datasets/yonatanbitton/SeeTRUE) \n\n+ [Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation](https://arxiv.org/abs/2305.01569) (2023-05-02) \n [](https://github.com/yuvalkirstain/PickScore)\n\n+ [Analysis of Appeal for Realistic AI-Generated Photos](https://ieeexplore.ieee.org/document/10103686) (2023-04-17) [](https://github.com/Telecommunication-Telemedia-Assessment/avt_ai_images)\n\n+ [Appeal and quality assessment for AI-generated images](https://ieeexplore.ieee.org/document/10178486) (2023-06-22) [](https://github.com/Telecommunication-Telemedia-Assessment/avt_ai_images)\n\n+ [Diagnostic Benchmark and Iterative Inpainting for Layout-Guided Image Generation](https://arxiv.org/abs/2304.06671) (2023-04-13) \n[](https://github.com/j-min/IterInpaint)\n[](https://layoutbench.github.io/)\n\n+ [HRS-Bench: Holistic, Reliable and Scalable Benchmark for Text-to-Image Models](https://arxiv.org/abs/2304.05390) (2023-04-11, ICCV 2023) \n [](https://github.com/eslambakr/HRS_benchmark) [](https://eslambakr.github.io/hrsbench.github.io/)\n\n+ [Human Preference Score: Better Aligning Text-to-Image Models with Human Preference](https://arxiv.org/abs/2303.14420) (2023-03-25, ICCV 2023) \n [](https://github.com/tgxs002/align_sd)\n [](https://tgxs002.github.io/align_sd_web/)\n\n+ [A study of the evaluation metrics for generative images containing combinational creativity](https://www-cambridge-org.remotexs.ntu.edu.sg/core/journals/ai-edam/article/study-of-the-evaluation-metrics-for-generative-images-containing-combinational-creativity/FBB623857EE474ED8CD2114450EA8484) (2023-03-23) \n >Note: Consensual Assessment Technique and Turing Test used in T2I evaluation\n\n+ [TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answering](https://arxiv.org/abs/2303.11897) (2023-03-21, ICCV 2023) \n [](https://github.com/Yushi-Hu/tifa) [](https://tifa-benchmark.github.io/)\n\n+ [Is This Loss Informative? Faster Text-to-Image Customization by Tracking Objective Dynamics](https://arxiv.org/abs/2302.04841) (2023-02-09) \n[](https://github.com/yandex-research/DVAR)\n >Note: an evaluation approach for early stopping criterion in T2I customization\n\n+ [Benchmarking Spatial Relationships in Text-to-Image Generation](https://arxiv.org/abs/2212.10015) (2022-12-20) \n [](https://github.com/microsoft/VISOR)\n\n+ MMI and MOR from from [Benchmarking Robustness of Multimodal Image-Text Models under Distribution Shift](https://arxiv.org/abs/2212.08044) (2022-12-15)\n[](https://mmrobustness.github.io/)\n\n+ [TeTIm-Eval: a novel curated evaluation data set for comparing text-to-image models](https://arxiv.org/abs/2212.07839) (2022-12-15) \n\n\n+ [Human Evaluation of Text-to-Image Models on a Multi-Task Benchmark](https://arxiv.org/abs/2211.12112) (2022-11-22)\n\n+ [UPainting: Unified Text-to-Image Diffusion Generation with Cross-modal Guidance](https://arxiv.org/abs/2210.16031) (2022-10-28)\n[](https://upainting.github.io/)\n > Note: UniBench, benchmark contains prompts for simple-scene images and complex-scene images in Chinese and English \n\n+ [Re-Imagen: Retrieval-Augmented Text-to-Image Generator](https://arxiv.org/abs/2209.14491) (2022-09-29)\n > Note: EntityDrawBench, benchmark to evaluates image generation for diverse entities\n\n+ [Vision-Language Matching for Text-to-Image Synthesis via Generative Adversarial Networks](https://arxiv.org/abs/2208.09596) (2022-08-20)\n > Note: new metric, Vision-Language Matching Score (VLMS)\n\n+ [Scaling Autoregressive Models for Content-Rich Text-to-Image Generation](https://arxiv.org/abs/2206.10789) (2022-06-22)\n [](https://github.com/google-research/parti) [](https://sites.research.google/parti/)\n\n+ [GR-GAN: Gradual Refinement Text-to-image Generation](https://arxiv.org/abs/2205.11273) (2022-05-23) \n[](https://github.com/BoO-18/GR-GAN)\n > Note: new metric Cross-Model Distance introduced \n\n+ [DrawBench from Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://arxiv.org/abs/2205.11487) (2022-05-23)\n[](https://imagen.research.google/)\n\n+ [StyleT2I: Toward Compositional and High-Fidelity Text-to-Image Synthesis](https://arxiv.org/abs/2203.15799) (2022-03-29, CVPR 2024)\n [](https://github.com/zhihengli-UR/StyleT2I)\n > Note: Evaluation metric for compositionality of T2I models\n\n\n+ [Benchmark for Compositional Text-to-Image Synthesis](https://openreview.net/forum?id=bKBhQhPeKaF) (2021-07-29) \n [](https://github.com/Seth-Park/comp-t2i-dataset)\n\n+ [TISE: Bag of Metrics for Text-to-Image Synthesis Evaluation](https://arxiv.org/abs/2112.01398) (2021-12-02, ECCV 2022) \n[](https://github.com/VinAIResearch/tise-toolbox)\n\n+ [Improving Generation and Evaluation of Visual Stories via Semantic Consistency](https://arxiv.org/abs/2105.10026) (2021-05-20)\n[](https://github.com/adymaharana/StoryViz)\n\n+ [Leveraging Visual Question Answering to Improve Text-to-Image Synthesis](https://arxiv.org/abs/2010.14953) (2020-10-28)\n\n+ [Image Synthesis from Locally Related Texts](https://dl.acm.org/doi/abs/10.1145/3372278.3390684) (2020-06-08)\n > Note: VQA accuracy as a new evaluation metric.\n\n+ [Semantic Object Accuracy for Generative Text-to-Image Synthesis](https://arxiv.org/abs/1910.13321) (2019-10-29) \n [](https://github.com/tohinz/semantic-object-accuracy-for-generative-text-to-image-synthesis?tab=readme-ov-file) [](https://www.tobiashinz.com/2019/10/30/semantic-object-accuracy-for-generative-text-to-image-synthesis)\n > Note: new evaluation metric, Semantic Object Accuracy (SOA)\n\n+ [GPT-ImgEval: A Comprehensive Benchmark for Diagnosing GPT4o in Image Generation](https://arxiv.org/abs/2504.02782) (2025-04-03)\n[](https://github.com/PicoTrex/GPT-ImgEval)\n\n+ [R2I-Bench: Benchmarking Reasoning-Driven Text-to-Image Generation](https://arxiv.org/abs/2505.23493) (2025-05-28)\n\n\n### 3.3. Evaluation of Text-Based Image Editing\n\n+ [Learning Action and Reasoning-Centric Image Editing from Videos and Simulations](https://arxiv.org/abs/2407.03471) (2024-07-03) \n > Note: AURORA-Bench introduced\n\n+ [GIM: A Million-scale Benchmark for Generative Image Manipulation Detection and Localization](https://arxiv.org/abs/2406.16531) (2024-06-24)\n[](https://github.com/chenyirui/GIM)\n\n\n+ [MultiEdits: Simultaneous Multi-Aspect Editing with Text-to-Image Diffusion Models](https://arxiv.org/abs/2406.00985) (2024-06-03)\n [](https://mingzhenhuang.com/projects/MultiEdits.html) [](https://huggingface.co/datasets/UB-CVML-Group/PIE_Bench_pp)\n > Note: PIE-Bench++, evaluating image-editing tasks involving multiple objects and attributes\n\n+ [DiffUHaul: A Training-Free Method for Object Dragging in Images](https://arxiv.org/abs/2406.01594) (2024-06-03)\n >Note: foreground similarity, object traces and realism metric introduced\n\n+ [HQ-Edit: A High-Quality Dataset for Instruction-based Image Editing](https://arxiv.org/abs/2404.09990) (2024-04-15) \n[](https://github.com/UCSC-VLAA/HQ-Edit)\n[](https://thefllood.github.io/HQEdit_web/) \n[](https://huggingface.co/datasets/UCSC-VLAA/HQ-Edit)\n\n+ [FlexEdit: Flexible and Controllable Diffusion-based Object-centric Image Editing](https://arxiv.org/abs/2403.18605) (2024-03-27) \n[](https://flex-edit.github.io/) \n >Note: novel automatic mask-based evaluation metric tailored to various object-centric editing scenarios\n\n+ TransformationOriented Paired Benchmark from [InstructBrush: Learning Attention-based Instruction Optimization for Image Editing](https://arxiv.org/abs/2403.18660) (2024-03-27)\n[](https://github.com/RoyZhao926/InstructBrush)\n [](https://royzhao926.github.io/InstructBrush/)\n\n+ ImageNet Concept Editing Benchmark from [Editing Massive Concepts in Text-to-Image Diffusion Models](https://arxiv.org/abs/2403.13807) (2024-03-20) \n[](https://github.com/SilentView/EMCID)\n[](https://silentview.github.io/EMCID/)\n\n+ [Editing Massive Concepts in Text-to-Image Diffusion Models](https://arxiv.org/abs/2403.13807) (2024-03-20)\n[](https://github.com/SilentView/EMCID) [](https://silentview.github.io/EMCID/)\n >Note: ImageNet Concept Editing Benchmark (ICEB), for evaluating massive concept editing for T2I models\n\n+ [Make Me Happier: Evoking Emotions Through Image Diffusion Models](https://arxiv.org/abs/2403.08255) (2024-03-13) \n >Note: EMR, ESR, ENRD, ESS metric introduced\n\n\n+ [Diffusion Model-Based Image Editing: A Survey](https://arxiv.org/abs/2402.17525) (2024-02-27) \n [](https://github.com/SiatMMLab/Awesome-Diffusion-Model-Based-Image-Editing-Methods)\n > Note: EditEval, benchmark for text-guided image editing and LLM Score\n\n+ [Towards Efficient Diffusion-Based Image Editing with Instant Attention Masks](https://arxiv.org/abs/2401.07709) (2024-01-15, AAAI 2024)\n[](https://github.com/xiaotianqing/InstDiffEdit)\n >Note: Editing-Mask, new benchmark to examine the mask accuracy and local editing ability\n\n+ [RotationDrag: Point-based Image Editing with Rotated Diffusion Features](https://arxiv.org/abs/2401.06442) (2024-01-12)\n[](https://github.com/Tony-Lowe/RotationDrag)\n >Note: RotationBench introduced\n\n+ [LEDITS++: Limitless Image Editing using Text-to-Image Models](https://arxiv.org/abs/2311.16711) (2023-11-28) \n [](https://huggingface.co/spaces/editing-images/leditsplusplus/tree/main) [](https://leditsplusplus-project.static.hf.space/index.html) [](https://huggingface.co/spaces/editing-images/leditsplusplus)\n > Note: TEdBench++, revised benchmark of TEdBench\n\n+ [Emu Edit: Precise Image Editing via Recognition and Generation Tasks](https://arxiv.org/abs/2311.10089) (2023-11-16)\n[](https://huggingface.co/datasets/facebook/emu_edit_test_set)\n[](https://emu-edit.metademolab.com/)\n\n\n+ [EditVal: Benchmarking Diffusion Based Text-Guided Image Editing Methods](https://arxiv.org/abs/2310.02426) (2023-10-03) \n [](https://github.com/deep-ml-research/editval_code)\n [](https://deep-ml-research.github.io/editval/)\n\n+ PIE-Bench from [Direct Inversion: Boosting Diffusion-based Editing with 3 Lines of Code](https://arxiv.org/abs/2310.01506) (2023-10-02) \n[](https://github.com/cure-lab/PnPInversion)\n [](https://cure-lab.github.io/PnPInversion/)\n\n+ [Iterative Multi-granular Image Editing using Diffusion Models](https://arxiv.org/abs/2309.00613) (2023-09-01)\n\n+ [DragDiffusion: Harnessing Diffusion Models for Interactive Point-based Image Editing](https://arxiv.org/abs/2306.14435) (2023-06-26) \n[](https://yujun-shi.github.io/projects/dragdiffusion.html)\n[](https://github.com/Yujun-Shi/DragDiffusion)\n > Note: drawbench benchmark introduced\n\n+ [DreamEdit: Subject-driven Image Editing](https://arxiv.org/abs/2306.12624) (2023-06-22)\n > Note: DreamEditBench benchmark introduced\n[](https://dreameditbenchteam.github.io/)\n[](https://github.com/DreamEditBenchTeam/DreamEdit)\n\n+ [MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editing](https://arxiv.org/abs/2306.10012) (2023-06-16) \n[](https://github.com/OSU-NLP-Group/MagicBrush)\n [](https://osu-nlp-group.github.io/MagicBrush/)\n [](https://huggingface.co/datasets/osunlp/MagicBrush)\n > Note: dataset only\n\n \n+ [Imagen Editor and EditBench: Advancing and Evaluating Text-Guided Image Inpainting](https://arxiv.org/abs/2212.06909) (2022-12-13, CVPR 2023) \n [](https://research.google/blog/imagen-editor-and-editbench-advancing-and-evaluating-text-guided-image-inpainting/)\n\n+ [Imagic: Text-Based Real Image Editing with Diffusion Models](https://arxiv.org/abs/2210.09276) (2022-10-17) \n [](https://github.com/imagic-editing/imagic-editing.github.io/tree/main/tedbench) [](https://imagic-editing.github.io/) [](https://huggingface.co/datasets/bahjat-kawar/tedbench)\n > Note: TEdBench, image editing benchmark\n\n+ [Predict, Prevent, and Evaluate: Disentangled Text-Driven Image Manipulation Empowered by Pre-Trained Vision-Language Model](https://arxiv.org/abs/2111.13333) (2021-11-26)\n[](https://github.com/zipengxuc/PPE)\n\n+ [Knowledge-Driven Generative Adversarial Network for Text-to-Image Synthesis](https://ieeexplore.ieee.org/abstract/document/9552559) (2021-09-29)\n [](https://github.com/pengjunn/KD-GAN)\n > Note: New evaluation system, Pseudo Turing Test (PTT)\n\n+ [ManiGAN: Text-Guided Image Manipulation](https://arxiv.org/abs/1912.06203) (2019-12-12) \n[](https://github.com/mrlibw/ManiGAN)\n >Note: manipulative precision metric introduced\n\n+ [Text Guided Person Image Synthesis](https://arxiv.org/abs/1904.05118) (2019-04-10)\n >Note: VQA perceptual score introduced\n \n\n### 3.4. Evaluation of Neural Style Transfer\n\n+ [ArtFID: Quantitative Evaluation of Neural Style Transfer](https://arxiv.org/abs/2207.12280) (2022-07-25)\n[](https://github.com/matthias-wright/art-fid)\n\n\n### 3.5. Evaluation of Video Generation\n\n#### 3.5.1. Evaluation of Text-to-Video Generation\n\n+ [Are Synthetic Videos Useful? A Benchmark for Retrieval-Centric Evaluation of Synthetic Videos](https://arxiv.org/abs/2507.02316) (2025-07-03) \n\n+ [AIGVE-MACS: Unified Multi-Aspect Commenting and Scoring Model for AI-Generated Video Evaluation](https://arxiv.org/abs/2507.01255) (2025-07-02) \n\n+ [BrokenVideos: A Benchmark Dataset for Fine-Grained Artifact Localization in AI-Generated Videos](https://arxiv.org/abs/2506.20103) (2025-06-25) \n\n+ [OpenS2V-Nexus: A Detailed Benchmark and Million-Scale Dataset for Subject-to-Video Generation](https://arxiv.org/abs/2505.20292) (2025-06-03)\n [](https://github.com/PKU-YuanGroup/OpenS2V-Nexus) \n [](https://pku-yuangroup.github.io/OpenS2V-Nexus/)\n >Note: The first open-sourced infrastructure (OpenS2V-Eval & OpenS2V-5M) for Subject-to-Video generation\n\n+ [LOVE: Benchmarking and Evaluating Text-to-Video Generation and Video-to-Text Interpretation](https://arxiv.org/abs/2505.12098) (2025-05-17)\n\n+ [On the Consistency of Video Large Language Models in Temporal Comprehension](https://arxiv.org/abs/2411.12951) (2025-05-17) \n\n+ [AIGVE-Tool: AI-Generated Video Evaluation Toolkit with Multifaceted Benchmark](https://arxiv.org/abs/2503.14064) (2025-04-18) \n\n+ [VideoGen-Eval: Agent-based System for Video Generation Evaluation](https://arxiv.org/abs/2503.23452) (2025-03-30) \n [](https://github.com/AILab-CVC/VideoGen-Eval)\n\n+ [Video-Bench: Human Preference Aligned Video Generation Benchmark](https://arxiv.org/abs/2504.04907) (2025-04-07) \n [](https://github.com/Video-Bench/Video-Bench)\n\n+ [Morpheus: Benchmarking Physical Reasoning of Video Generative Models with Real Physical Experiments](https://arxiv.org/abs/2504.02918) (2025-04-03)\n\n+ [Envisioning Beyond the Pixels: Benchmarking Reasoning-Informed Visual Editing](https://arxiv.org/abs/2504.02826) (2025-04-03)\n[](https://github.com/PhoenixZ810/RISEBench)\n\n+ [VinaBench: Benchmark for Faithful and Consistent Visual Narratives](https://arxiv.org/abs/2503.20871) (2025-03-26)\n[](https://github.com/Silin159/VinaBench)\n\n+ [ETVA: Evaluation of Text-to-Video Alignment via Fine-grained Question Generation and Answering](https://arxiv.org/abs/2503.16867) (2025-03-21)\n+ [Is Your World Simulator a Good Story Presenter? A Consecutive Events-Based Benchmark for Future Long Video Generation](https://arxiv.org/abs/2412.16211) (2024-12-17)\n >Note: focus on storytelling. \n\n+ [Evaluation Agent: Efficient and Promptable Evaluation Framework for Visual Generative Models](https://arxiv.org/abs/2412.09645) (2024-12-16)\n [](https://github.com/Vchitect/Evaluation-Agent) \n [](https://vchitect.github.io/Evaluation-Agent-project/)\n >Note: focus on efficient and dynamic evaluation. \n\n+ [Neuro-Symbolic Evaluation of Text-to-Video Models using Formal Verification](https://arxiv.org/abs/2411.16718) (2024-12-03)\n >Note: focus on temporally text-video alignment (event order, accuracy)\n\n+ [AIGV-Assessor: Benchmarking and Evaluating the Perceptual Quality of Text-to-Video Generation with LMM](https://arxiv.org/abs/2411.17221) (2024-11-26)\n [](https://github.com/wangjiarui153/AIGV-Assessor) \n >Note: fuild motion, light change, motion speed, event order. \n\n+ [What You See Is What Matters: A Novel Visual and Physics-Based Metric for Evaluating Video Generation Quality](https://arxiv.org/abs/2411.13609) (2024-11-24)\n >Note: texture evaluation scheme introduced\n\n+ [A Survey of AI-Generated Video Evaluation](https://arxiv.org/abs/2410.19884) (2024-10-24)\n\n+ [The Dawn of Video Generation: Preliminary Explorations with SORA-like Models](https://arxiv.org/abs/2410.05227) (2024-10-10)\n\n\n+ [Towards World Simulator: Crafting Physical Commonsense-Based Benchmark for Video Generation](https://arxiv.org/abs/2410.05363) (2024-10-07)\n [](https://github.com/OpenGVLab/PhyGenBench) \n [](https://phygenbench123.github.io/)\n >Note: Comprehensive physical (optical, mechanic, thermal, material) benchmark introduced\n\n+ [Benchmarking AIGC Video Quality Assessment: A Dataset and Unified Model](https://arxiv.org/abs/2407.21408) (2024-07-31)\n\n\n+ [T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video Generation](https://arxiv.org/abs/2407.14505) (2024-07-19) \n [](https://github.com/KaiyueSun98/T2V-CompBench) \n [](https://t2v-compbench.github.io/)\n\n\n+ [T2VSafetyBench: Evaluating the Safety of Text-to-Video Generative Models](https://arxiv.org/abs/2407.05965) (2024-07-08)\n >Note: T2VSafetyBench introduced\n\n\n\n+ [Evaluation of Text-to-Video Generation Models: A Dynamics Perspective](https://arxiv.org/abs/2407.01094) (2024-07-01)\n\n\n+ [T2VBench: Benchmarking Temporal Dynamics for Text-to-Video Generation](https://openaccess.thecvf.com/content/CVPR2024W/EvGenFM/html/Ji_T2VBench_Benchmarking_Temporal_Dynamics_for_Text-to-Video_Generation_CVPRW_2024_paper.html) (2024-06)\n\n\n+ [Evaluating and Improving Compositional Text-to-Visual Generation](https://openaccess.thecvf.com/content/CVPR2024W/EvGenFM/html/Li_Evaluating_and_Improving_Compositional_Text-to-Visual_Generation_CVPRW_2024_paper.html) (2024-06)\n\n\n+ [TlTScore: Towards Long-Tail Effects in Text-to-Visual Evaluation with Generative Foundation Models](https://openaccess.thecvf.com/content/CVPR2024W/EvGenFM/html/Ji_TlTScore_Towards_Long-Tail_Effects_in_Text-to-Visual_Evaluation_with_Generative_Foundation_CVPRW_2024_paper.html) (2024-06)\n\n\n+ [ChronoMagic-Bench: A Benchmark for Metamorphic Evaluation of Text-to-Time-lapse Video Generation](https://arxiv.org/abs/2406.18522) (2024-06-26)\n [](https://github.com/PKU-YuanGroup/ChronoMagic-Bench)\n [](https://pku-yuangroup.github.io/ChronoMagic-Bench/)\n [](https://huggingface.co/spaces/BestWishYsh/ChronoMagic-Bench)\n >Note: Comprehensive time-lapse (biological, human created, meteorological, physical) benchmark introduced\n \n+ [VideoScore: Building Automatic Metrics to Simulate Fine-grained Human Feedback for Video Generation](https://arxiv.org/abs/2406.15252) (2024-06-21)\n[](https://github.com/TIGER-AI-Lab/VideoScore)\n[](https://huggingface.co/datasets/TIGER-Lab/VideoFeedback)\n\n+ [TC-Bench: Benchmarking Temporal Compositionality in Text-to-Video and Image-to-Video Generation](https://arxiv.org/abs/2406.08656) (2024-06-12)\n >Note: TC-Bench, TCR and TC-Score introduced\n\n+ [VideoPhy: Evaluating Physical Commonsense for Video Generation](https://arxiv.org/abs/2406.03520v1) (2024-06-05) \n [](https://videophy.github.io)\n [](https://github.com/Hritikbansal/videophy)\n\n+ [Illumination Histogram Consistency Metric for Quantitative Assessment of Video Sequences](https://arxiv.org/abs/2405.09716) (2024-05-15)\n[](https://github.com/LongChenCV/IHC-Metric)\n\n+ [The Lost Melody: Empirical Observations on Text-to-Video Generation From A Storytelling Perspective](https://arxiv.org/abs/2405.08720) (2024-05-13) \n > Note: New evaluation framework T2Vid2T, Evaluation for storytelling aspects of videos\n\n+ [Exposing AI-generated Videos: A Benchmark Dataset and a Local-and-Global Temporal Defect Based Detection Method](https://arxiv.org/abs/2405.04133) (2024-05-07) \n\n+ [Sora Detector: A Unified Hallucination Detection for Large Text-to-Video Models](https://arxiv.org/abs/2405.04180) (2024-05-07) \n[](https://bytez.com/docs/arxiv/2405.04180/llm)\n > Note: hallucination detection\n\n+ [Exploring AIGC Video Quality: A Focus on Visual Harmony, Video-Text Consistency and Domain Distribution Gap](https://arxiv.org/abs/2404.13573) (2024-04-21) \n [](https://github.com/Coobiw/TriVQA)\n\n+ [Subjective-Aligned Dataset and Metric for Text-to-Video Quality Assessment](https://arxiv.org/abs/2403.11956) (2024-03-18) \n [](https://github.com/QMME/T2VQA)\n\n+ [A dataset of text prompts, videos and video quality metrics from generative text-to-video AI models](https://www.sciencedirect.com/science/article/pii/S2352340924004839) (2024-02-22) \n [](https://github.com/Chiviya01/Evaluating-Text-to-Video-Models)\n\n+ [Sora Generates Videos with Stunning Geometrical Consistency](https://arxiv.org/abs/2402.17403) (2024-02-27) \n [](https://github.com/meteorshowers/Sora-Generates-Videos-with-Stunning-Geometrical-Consistency)\n [](https://sora-geometrical-consistency.github.io)\n\n\n+ [STREAM: Spatio-TempoRal Evaluation and Analysis Metric for Video Generative Models](https://arxiv.org/abs/2403.09669) (2024-01-30) \n [](https://github.com/pro2nit/STREAM)\n\n\n+ [Towards A Better Metric for Text-to-Video Generation](https://arxiv.org/abs/2401.07781) (2024-01-15) \n [](https://github.com/showlab/T2VScore) [](https://showlab.github.io/T2VScore/) [](https://huggingface.co/datasets/jayw/t2v-gen-eval)\n\n+ [PEEKABOO: Interactive Video Generation via Masked-Diffusion](https://arxiv.org/abs/2312.07509) (2023-12-12) \n [](https://github.com/microsoft/Peekaboo)\n > Note: Benchmark for interactive video generation \n\n+ [VBench: Comprehensive Benchmark Suite for Video Generative Models](https://arxiv.org/abs/2311.17982) (2023-11-29) \n [](https://github.com/Vchitect/VBench) [](https://vchitect.github.io/VBench-project/) [](https://huggingface.co/spaces/Vchitect/VBench_Leaderboard)\n\n+ [SmoothVideo: Smooth Video Synthesis with Noise Constraints on Diffusion Models for One-shot Video Tuning](https://arxiv.org/abs/2311.17536) (2023-11-29) \n[](https://github.com/SPengLiang/SmoothVideo)\n\n\n+ [FETV: A Benchmark for Fine-Grained Evaluation of Open-Domain Text-to-Video Generation](https://arxiv.org/abs/2311.01813) (2023-11-03) \n [](https://github.com/llyx97/FETV)\n\n+ [EvalCrafter: Benchmarking and Evaluating Large Video Generation Models](https://arxiv.org/abs/2310.11440) (2023-10-17) \n [](https://github.com/EvalCrafter/EvalCrafter)\n [](https://evalcrafter.github.io) [](https://huggingface.co/datasets/RaphaelLiu/EvalCrafter_T2V_Dataset) [](https://huggingface.co/spaces/AILab-CVC/EvalCrafter)\n\n+ [Measuring the Quality of Text-to-Video Model Outputs: Metrics and Dataset](https://arxiv.org/abs/2309.08009) (2023-09-14)\n\n+ [StoryBench: A Multifaceted Benchmark for Continuous Story Visualization](https://arxiv.org/abs/2308.11606) (2023-08-22, NeurIPS 2023) \n [](https://github.com/google/storybench)\n\n+ [Exploring Video Quality Assessment on User Generated Contents from Aesthetic and Technical Perspectives](https://arxiv.org/abs/2211.04894) (2023-03-07, ICCV 2023) \n [](https://github.com/VQAssessment/DOVER)\n > Note: Aesthetic View & Technical View\n\n+ [CelebV-Text: A Large-Scale Facial Text-Video Dataset](https://arxiv.org/abs/2303.14717) (2023-03-26, CVPR 2023) \n [](https://github.com/CelebV-Text/CelebV-Text) [](https://celebv-text.github.io/) \n > Note: Benchmark on Facial Text-to-Video Generation\n\n+ [Make It Move: Controllable Image-to-Video Generation with Text Descriptions](https://arxiv.org/abs/2112.02815) (2021-12-06, CVPR 2022) \n [](https://github.com/Youncy-Hu/MAGE)\n \n\n#### 3.5.2. Evaluation of Image-to-Video Generation\n\n+ [VBench++: Comprehensive and Versatile Benchmark Suite for Video Generative Models](https://arxiv.org/abs/2411.13503) (2024-11-20) \n [](https://github.com/Vchitect/VBench/tree/master/vbench2_beta_i2v)\n [](https://vchitect.github.io/VBench-project/)\n\n+ I2V-Bench from [ConsistI2V: Enhancing Visual Consistency for Image-to-Video Generation](https://arxiv.org/abs/2402.04324) (2024-02-06) \n [](https://github.com/TIGER-AI-Lab/ConsistI2V) [](https://tiger-ai-lab.github.io/ConsistI2V/) [](https://huggingface.co/spaces/TIGER-Lab/ConsistI2V)\n\n+ [AIGCBench: Comprehensive Evaluation of Image-to-Video Content Generated by AI](https://arxiv.org/abs/2401.01651) (2024-01-03) \n [](https://github.com/BenchCouncil/AIGCBench)\n [](https://www.benchcouncil.org/AIGCBench/) [](https://huggingface.co/datasets/stevenfan/AIGCBench_v1.0)\n\n\n+ [A Benchmark for Controllable Text-Image-to-Video Generation](https://ieeexplore.ieee.org/abstract/document/10148799) (2023-06-12)\n\n+ [Temporal Shift GAN for Large Scale Video Generation](https://arxiv.org/abs/2004.01823) (2020-04-04) \n[](https://github.com/amunozgarza/tsb-gan)\n >Note: Symmetric-Similarity-Score introduced\n\n+ [Video Imagination from a Single Image with Transformation Generation](https://arxiv.org/abs/1706.04124) (2017-06-13) \n >Note: RIQA metric introduced\n\n\n#### 3.5.3. Evaluation of Talking Face Generation\n\n+ [OpFlowTalker: Realistic and Natural Talking Face Generation via Optical Flow Guidance](https://arxiv.org/abs/2405.14709) (2024-05-23)\n > Note: VTCS to measures lip-readability in synthesized videos\n\n+ [Audio-Visual Speech Representation Expert for Enhanced Talking Face Video Generation and Evaluation](https://arxiv.org/abs/2405.04327) (2024-05-07) \n\n+ [VASA-1: Lifelike Audio-Driven Talking Faces Generated in Real Time](https://arxiv.org/abs/2404.10667) (2024-04-16)\n[](https://www.microsoft.com/en-us/research/project/vasa-1/)\n >Note: Contrastive Audio and Pose Pretraining (CAPP) score introduced\n\n+ [THQA: A Perceptual Quality Assessment Database for Talking Heads](https://arxiv.org/abs/2404.09003) (2024-04-13) \n [](https://github.com/zyj-2000/THQA)\n\n+ [A Comparative Study of Perceptual Quality Metrics for Audio-driven Talking Head Videos](https://arxiv.org/abs/2403.06421) (2024-03-11) \n [](https://github.com/zwx8981/ADTH-QA)\n\n+ [Seeing What You Said: Talking Face Generation Guided by a Lip Reading Expert](https://arxiv.org/abs/2303.17480) (2023-03-29, CVPR 2023) \n [](https://github.com/Sxjdwang/TalkLip)\n > Note: Measuring intelligibility of the generated videos\n\n+ [Sparse in Space and Time: Audio-visual Synchronisation with Trainable Selectors](https://arxiv.org/abs/2210.07055) (2022-10-13) \n [](https://github.com/v-iashin/SparseSync)\n\n+ [Responsive Listening Head Generation: A Benchmark Dataset and Baseline](https://arxiv.org/abs/2112.13548) (2021-12-27, ECCV 2022) \n [](https://github.com/dc3ea9f/vico_challenge_baseline) [](https://project.mhzhou.com/vico/)\n\n+ [A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wild](https://arxiv.org/abs/2008.10010) (2020-08-23) \n >Note: new metric LSE-D and LSE-C introduced\n\n+ [What comprises a good talking-head video generation?: A Survey and Benchmark](https://arxiv.org/abs/2005.03201) (2020-05-07) \n[](https://github.com/lelechen63/talking-head-generation-survey)\n\n#### 3.5.4. Evaluation of World Generation\n\n+ [WorldScore: A Unified Evaluation Benchmark for World Generation](https://arxiv.org/abs/2504.00983) (2025-04-01)\n [](https://github.com/haoyi-duan/WorldScore) [](https://haoyi-duan.github.io/WorldScore/)\n\n\n### 3.6. Evaluation of Text-to-Motion Generation\n\n+ [VMBench: A Benchmark for Perception-Aligned Video Motion Generation](https://arxiv.org/abs/2503.10076) (2024-03-13) \n\n+ [MoDiPO: text-to-motion alignment via AI-feedback-driven Direct Preference Optimization](https://arxiv.org/abs/2405.03803) (2024-05-06) \n\n+ [What is the Best Automated Metric for Text to Motion Generation?](https://arxiv.org/abs/2309.10248) (2023-09-19) \n\n+ [Text-to-Motion Retrieval: Towards Joint Understanding of Human Motion Data and Natural Language](https://arxiv.org/abs/2305.15842) (2023-05-25) \n[](https://github.com/mesnico/text-to-motion-retrieval)\n > Note: Evaluation protocol for assessing the quality of the retrieved motions\n\n+ [Establishing a Unified Evaluation Framework for Human Motion Generation: A Comparative Analysis of Metrics](https://arxiv.org/abs/2405.07680) (2024-05-13) \n[](https://github.com/MSD-IRIMAS/Evaluating-HMG)\n\n+ [Evaluation of text-to-gesture generation model using convolutional neural network](https://www.sciencedirect.com/science/article/pii/S0893608022001198) (2021-10-11)\n[](https://github.com/GestureGeneration/text2gesture_cnn)\n\n\n\n### 3.7. Evaluation of Model Trustworthiness\n\n#### 3.7.1. Evaluation of Visual-Generation-Model Trustworthiness\n\n+ [Bias in Gender Bias Benchmarks: How Spurious Features Distort Evaluation](https://arxiv.org/abs/2509.07596) (2025-09-09)\n\n+ [MLLM-as-a-Judge for Image Safety without Human Labeling](https://arxiv.org/abs/2501.00192) (2024-12-31) \n\n+ [VBench++: Comprehensive and Versatile Benchmark Suite for Video Generative Models](https://arxiv.org/abs/2411.13503) (2024-11-20) \n [](https://github.com/Vchitect/VBench/tree/master/vbench2_beta_trustworthiness)\n [](https://vchitect.github.io/VBench-project/)\n\n+ [BIGbench: A Unified Benchmark for Social Bias in Text-to-Image Generative Models Based on Multi-modal LLM](https://arxiv.org/abs/2407.15240) (2024-07-21) \n [](https://github.com/BIGbench2024/BIGbench2024/)\n\n\n+ [Towards Understanding Unsafe Video Generation](https://arxiv.org/abs/2407.12581) (2024-07-17)\n >Note: Proposes Latent Variable Defense (LVD) which works within the model's internal sampling process\n\n\n+ [The Factuality Tax of Diversity-Intervened Text-to-Image Generation: Benchmark and Fact-Augmented Intervention](https://arxiv.org/abs/2407.00377) (2024-06-29)\n\n\n+ [FairCoT: Enhancing Fairness in Text-to-Image Generation via Chain of Thought Reasoning with Multimodal Large Language Models](https://arxiv.org/abs/2406.09070) (2024-06-13)\n >Note: Normalized Entropy metric introduced\n\n+ [Latent Directions: A Simple Pathway to Bias Mitigation in Generative AI](https://arxiv.org/abs/2406.06352) (2024-06-10)\n[](https://github.com/blclo/latent-debiasing-directions) [](https://latent-debiasing-directions.compute.dtu.dk/)\n\n+ [Evaluating and Mitigating IP Infringement in Visual Generative AI](https://arxiv.org/abs/2406.04662) (2024-06-07)\n[](https://github.com/ZhentingWang/GAI_IP_Infringement)\n\n+ [Improving Geo-diversity of Generated Images with Contextualized Vendi Score Guidance](https://arxiv.org/abs/2406.04551) (2024-06-06)\n\n+ [AI-Face: A Million-Scale Demographically Annotated AI-Generated Face Dataset and Fairness Benchmark](https://arxiv.org/abs/2406.00783) (2024-06-02) \n [](https://github.com/Purdue-M2/AI-Face-FairnessBench)\n\n+ [FAIntbench: A Holistic and Precise Benchmark for Bias Evaluation in Text-to-Image Models](https://arxiv.org/abs/2405.17814) (2024-05-28)\n\n+ [ART: Automatic Red-teaming for Text-to-Image Models to Protect Benign Users](https://arxiv.org/abs/2405.19360) (2024-05-24) \n\n+ Condition Likelihood Discrepancy from [Membership Inference on Text-to-Image Diffusion Models via Conditional Likelihood Discrepancy](https://arxiv.org/abs/2405.14800) (2024-05-23)\n\n+ [Could It Be Generated? Towards Practical Analysis of Memorization in Text-To-Image Diffusion Models](https://arxiv.org/abs/2405.05846) (2024-05-09) \n\n\n+ [Towards Geographic Inclusion in the Evaluation of Text-to-Image Models](https://arxiv.org/abs/2405.04457) (2024-05-07) \n\n+ [UnsafeBench: Benchmarking Image Safety Classifiers on Real-World and AI-Generated Images](https://arxiv.org/abs/2405.03486) (2024-05-06) \n\n+ [Espresso: Robust Concept Filtering in Text-to-Image Models](https://arxiv.org/abs/2404.19227) (2024-04-30) \n > Note: Paper is about filtering unacceptable concepts, not evaluation.\n\n+ [Ethical-Lens: Curbing Malicious Usages of Open-Source Text-to-Image Models](https://arxiv.org/abs/2404.12104) (2024-04-18) \n [](https://github.com/yuzhu-cai/Ethical-Lens)\n\n+ [OpenBias: Open-set Bias Detection in Text-to-Image Generative Models](https://arxiv.org/abs/2404.07990) (2024-04-11) \n[](https://github.com/Picsart-AI-Research/OpenBias)\n\n+ [Survey of Bias In Text-to-Image Generation: Definition, Evaluation, and Mitigation](https://arxiv.org/abs/2404.01030) (2024-04-01)\n\n+ [Lost in Translation? Translation Errors and Challenges for Fair Assessment of Text-to-Image Models on Multilingual Concepts](https://arxiv.org/abs/2403.11092) (2024-03-17, NAACL 2024)\n\n+ [Evaluating Text-to-Image Generative Models: An Empirical Study on Human Image Synthesis](https://arxiv.org/abs/2403.05125) (2024-03-08) \n\n+ [Position: Towards Implicit Prompt For Text-To-Image Models](https://arxiv.org/abs/2403.02118) (2024-03-04)\n >Note: ImplicitBench, new benchmark\n\n\n+ [The Male CEO and the Female Assistant: Probing Gender Biases in Text-To-Image Models Through Paired Stereotype Test](https://arxiv.org/abs/2402.11089) (2024-02-16) \n\n+ [Multilingual Text-to-Image Generation Magnifies Gender Stereotypes and Prompt Engineering May Not Help You](https://arxiv.org/abs/2401.16092) (2024-01-29) \n[](https://github.com/felifri/magbig)\n\n+ [Benchmarking the Fairness of Image Upsampling Methods](https://arxiv.org/abs/2401.13555) (2024-01-24) \n\n+ [ViSAGe: A Global-Scale Analysis of Visual Stereotypes in Text-to-Image Generation](https://arxiv.org/abs/2401.06310) (2024-01-02)\n\n+ [New Job, New Gender? Measuring the Social Bias in Image Generation Models](https://arxiv.org/abs/2401.00763) (2024-01-01)\n\n+ Distribution Bias, Jaccard Hallucination, Generative Miss Rate from [Quantifying Bias in Text-to-Image Generative Models](https://arxiv.org/abs/2312.13053) (2023-12-20)\n[](https://huggingface.co/spaces/JVice/try-before-you-bias)\n[](https://github.com/JJ-Vice/TryBeforeYouBias)\n\n+ [TIBET: Identifying and Evaluating Biases in Text-to-Image Generative Models](https://arxiv.org/abs/2312.01261) (2023-12-03) \n >Note: CAS and BAV novel metric introduced\n\n+ [Holistic Evaluation of Text-To-Image Models](https://arxiv.org/abs/2311.04287) (2023-11-07) \n [](https://github.com/stanford-crfm/helm)\n [](https://crfm.stanford.edu/helm/heim/v1.1.0/)\n\n+ [Sociotechnical Safety Evaluation of Generative AI Systems](https://arxiv.org/abs/2310.11986) (2023-10-18) \n[](https://deepmind.google/discover/blog/evaluating-social-and-ethical-risks-from-generative-ai/)\n\n+ [Navigating Cultural Chasms: Exploring and Unlocking the Cultural POV of Text-To-Image Models](https://arxiv.org/abs/2310.01929) (2023-10-03)\n > Note: Evaluate the cultural content of TTI-generated images \n\n+ [ITI-GEN: Inclusive Text-to-Image Generation](https://arxiv.org/abs/2309.05569) (2023-09-11, ICCV 2023) \n [](https://czhang0528.github.io/iti-gen) \n [](https://github.com/humansensinglab/ITI-GEN) \n\n+ [DIG In: Evaluating Disparities in Image Generations with Indicators for Geographic Diversity](https://arxiv.org/abs/2308.06198) (2023-08-11) \n [](https://github.com/facebookresearch/DIG-In)\n\n+ [On the Cultural Gap in Text-to-Image Generation](https://arxiv.org/abs/2307.02971) (2023-07-06)\n[](https://github.com/longyuewangdcu/C3-Bench)\n\n+ [Evaluating the Robustness of Text-to-image Diffusion Models against Real-world Attacks](https://arxiv.org/abs/2306.13103) (2023-06-16) \n\n+ [Disparities in Text-to-Image Model Concept Possession Across Languages](https://dl.acm.org/doi/abs/10.1145/3593013.3594123) (2023-06-12)\n > Note: Benchmark of multilingual parity in conceptual possession\n\n+ [Evaluating the Social Impact of Generative AI Systems in Systems and Society](https://arxiv.org/abs/2306.05949) (2023-06-09) \n\n+ [Word-Level Explanations for Analyzing Bias in Text-to-Image Models](https://arxiv.org/abs/2306.05500) (2023-06-03) \n\n+ [Multilingual Conceptual Coverage in Text-to-Image Models](https://arxiv.org/abs/2306.01735) (2023-06-02, ACL 2023) \n [](https://github.com/michaelsaxon/CoCoCroLa) \n [](https://saxon.me/coco-crola/)\n > Note: CoCo-CroLa, benchmark for multilingual parity of text-to-image models\n\n+ [T2IAT: Measuring Valence and Stereotypical Biases in Text-to-Image Generation](https://arxiv.org/abs/2306.00905) (2023-06-01) \n[](https://github.com/eric-ai-lab/T2IAT) \n\n+ [SneakyPrompt: Jailbreaking Text-to-image Generative Models](https://arxiv.org/abs/2305.12082) (2023-05-20) \n [](https://github.com/Yuchen413/text2image_safety)\n\n+ [Inspecting the Geographical Representativeness of Images from Text-to-Image Models](https://arxiv.org/abs/2305.11080) (2023-05-18)\n\n+ [Multimodal Composite Association Score: Measuring Gender Bias in Generative Multimodal Models](https://arxiv.org/abs/2304.13855) (2023-04-26) \n\n+ [Uncurated Image-Text Datasets: Shedding Light on Demographic Bias](https://arxiv.org/abs/2304.02828) (2023-04-06, CVPR 2023) \n [](https://github.com/noagarcia/phase)\n\n+ [Social Biases through the Text-to-Image Generation Lens](https://arxiv.org/abs/2304.06034) (2023-03-30) \n\n\n+ [Stable Bias: Analyzing Societal Representations in Diffusion Models](https://arxiv.org/abs/2303.11408) (2023-03-20)\n\n\n\n+ [Auditing Gender Presentation Differences in Text-to-Image Models](https://arxiv.org/abs/2302.03675) (2023-02-07)\n [](https://github.com/SALT-NLP/GEP_data) [](https://salt-nlp.github.io/GEP/)\n\n+ [Towards Equitable Representation in Text-to-Image Synthesis Models with the Cross-Cultural Understanding Benchmark (CCUB) Dataset](https://arxiv.org/abs/2301.12073) (2023-01-28) \n\n+ [Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Models](https://arxiv.org/abs/2211.05105) (2022-11-09, CVPR 2023) \n[](https://github.com/ml-research/safe-latent-diffusion?tab=readme-ov-file) \n > Note: SLD removes and suppresses inappropriate image parts during the diffusion process\n\n+ [How well can Text-to-Image Generative Models understand Ethical Natural Language Interventions?](https://arxiv.org/abs/2210.15230) (2022-10-27)\n[](https://github.com/j-min/DallEval)\n\n+ [Exploiting Cultural Biases via Homoglyphs in Text-to-Image Synthesis](https://arxiv.org/abs/2209.08891) (2022-09-19)\n [](https://github.com/LukasStruppek/Exploiting-Cultural-Biases-via-Homoglyphs)\n\n\n+ [DALL-Eval: Probing the Reasoning Skills and Social Biases of Text-to-Image Generation Models](https://arxiv.org/abs/2202.04053) (2022-02-08, ICCV 2023) \n [](https://github.com/Hritikbansal/entigen_emnlp)\n > Note: PaintSkills, evaluation for visual reasoning capabilities and social biases\n\n#### 3.7.2. Evaluation of Non-Visual-Generation-Model Trustworthiness\nNot for visual generation, but related evaluations of other models like LLMs\n\n+ [The African Woman is Rhythmic and Soulful: Evaluation of Open-ended Generation for Implicit Biases](https://arxiv.org/abs/2407.01270) (2024-07-01) \n\n+ [Extrinsic Evaluation of Cultural Competence in Large Language Models](https://arxiv.org/abs/2406.11565) (2024-06-17)\n\n+ [Benchmarking Trustworthiness of Multimodal Large Language Models: A Comprehensive Study](https://arxiv.org/abs/2406.07057) (2024-06-11) \n\n+ [HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal](https://arxiv.org/abs/2402.04249) (2024-02-06) \n [](https://github.com/centerforaisafety/HarmBench)\n [](https://www.harmbench.org)\n\n+ [FACET: Fairness in Computer Vision Evaluation Benchmark](https://arxiv.org/abs/2309.00035) (2023-08-31) \n[](https://ai.meta.com/research/publications/facet-fairness-in-computer-vision-evaluation-benchmark/)\n[](https://facet.metademolab.com/)\n\n\n+ [Gender Biases in Automatic Evaluation Metrics for Image Captioning](https://arxiv.org/abs/2305.14711) (2023-05-24) \n\n+ [Fairness Indicators for Systematic Assessments of Visual Feature Extractors](https://arxiv.org/abs/2202.07603) (2022-02-15)\n[](https://github.com/facebookresearch/vissl/tree/main/projects/fairness_indicators)\n[](https://ai.meta.com/blog/meta-ai-research-explores-new-public-fairness-benchmarks-for-computer-vision-models/)\n\n\n\n### 3.8. Evaluation of Entity Relation\n\n+ Scene Graph(SG)-IoU, Relation-IoU, and Entity-IoU (using GPT-4v) from [SG-Adapter: Enhancing Text-to-Image Generation with Scene Graph Guidance](https://arxiv.org/abs/2405.15321) (2024-05-24)\n\n+ Relation Accuracy & Entity Accuracy from [ReVersion: Diffusion-Based Relation Inversion from Images](https://arxiv.org/abs/2303.13495) (2023-03-23)\n[](https://github.com/ziqihuangg/ReVersion)\n[](https://ziqihuangg.github.io/projects/reversion.html)\n[](https://huggingface.co/spaces/Ziqi/ReVersion)\n\n+ [Testing Relational Understanding in Text-Guided Image Generation](https://arxiv.org/abs/2208.00005) (2022-07-29) \n\n\n### 3.9. Agentic Evaluation\n\n+ [A Unified Agentic Framework for Evaluating Conditional Image Generation](https://arxiv.org/abs/2504.07046) (2025-04-09)\n[](https://github.com/HITsz-TMG/Agentic-CIGEval)\n\n+ [Evaluation Agent: Efficient and Promptable Evaluation Framework for Visual Generative Models](https://arxiv.org/abs/2412.09645) (2024-12-10)\n[](https://github.com/Vchitect/Evaluation-Agent)\n[](https://vchitect.github.io/Evaluation-Agent-project/)\n\n+ [VideoGen-Eval: Agent-based System for Video Generation Evaluation](https://arxiv.org/abs/2503.23452) (2025-03-30)\n[](https://github.com/AILab-CVC/VideoGen-Eval)\n[](https://ailab-cvc.github.io/VideoGen-Eval/)\n\n+ [Evaluating Hallucination in Text-to-Image Diffusion Models with Scene-Graph based Question-Answering Agent](https://arxiv.org/abs/2412.05722) (2024-12-07)\n\n\n\n## 4. Improving Visual Generation with Evaluation / Feedback / Reward\n\n+ [OneReward: Unified Mask-Guided Image Generation via Multi-Task Human Preference Learning](https://arxiv.org/abs/2508.21066) (2025-08-28)[](https://github.com/bytedance/OneReward) [](https://one-reward.github.io)\n\n+ [Prompt-A-Video: Prompt Your Video Diffusion Model via Preference-Aligned LLM](https://arxiv.org/abs/2412.15156) (2024-12-19) [](https://github.com/jiyt17/Prompt-A-Video)\n\n+ [Improved video generation with human feedback](https://arxiv.org/pdf/2501.13918) (2025-01-23) [](https://gongyeliu.github.io/videoalign/)\n\n+ [LiFT: Leveraging Human Feedback for Text-to-Video Model Alignment](https://arxiv.org/pdf/2412.04814) (2024-12-24) []() [](https://codegoat24.github.io/LiFT/)\n\n+ [VideoDPO: Omni-Preference Alignment for Video Diffusion Generation](https://arxiv.org/abs/2412.14167) (2024-12-18) []() [](https://github.com/CIntellifusion/VideoDPO)\n\n+ [Boosting Text-to-Video Generative Model with MLLMs Feedback](https://openreview.net/pdf/4c9eebaad669788792e0a010be4031be5bdc426e.pdf) (2024-09-26,NeurIPS 2024)\n\n+ [Direct Unlearning Optimization for Robust and Safe Text-to-Image Models](https://arxiv.org/abs/2407.21035) (2024-07-17) \n\n+ [Safeguard Text-to-Image Diffusion Models with Human Feedback Inversion](https://arxiv.org/abs/2407.21032) (2024-07-17, ECCV 2024) \n\n+ [Subject-driven Text-to-Image Generation via Preference-based Reinforcement Learning](https://arxiv.org/abs/2407.12164) (2024-07-16)\n\n+ [Video Diffusion Alignment via Reward Gradients](https://arxiv.org/abs/2407.08737) (2024-07-11)\n[](https://github.com/mihirp1998/VADER) [](https://vader-vid.github.io/)\n\n\n\n+ [Powerful and Flexible: Personalized Text-to-Image Generation via Reinforcement Learning](https://arxiv.org/abs/2407.06642) (2024-07-09) \n[](https://github.com/wfanyue/DPG-T2I-Personalization)\n\n+ [Aligning Human Motion Generation with Human Perceptions](https://arxiv.org/abs/2407.02272) (2024-07-02) \n[](https://github.com/ou524u/AlignHP)\n\n\n+ [PopAlign: Population-Level Alignment for Fair Text-to-Image Generation](https://arxiv.org/abs/2406.19668) (2024-06-28)\n[](https://github.com/jacklishufan/PopAlignSDXL)\n\n\n+ [Prompt Refinement with Image Pivot for Text-to-Image Generation](https://arxiv.org/abs/2407.00247) (2024-06-28, ACL 2024)\n\n+ [Diminishing Stereotype Bias in Image Generation Model using Reinforcemenlent Learning Feedback](https://arxiv.org/abs/2407.09551) (2024-06-27)\n\n+ [Beyond Thumbs Up/Down: Untangling Challenges of Fine-Grained Feedback for Text-to-Image Generation](https://arxiv.org/abs/2406.16807) (2024-06-24)\n\n\n\n+ [Batch-Instructed Gradient for Prompt Evolution: Systematic Prompt Optimization for Enhanced Text-to-Image Synthesis](https://arxiv.org/abs/2406.08713) (2024-06-13)\n\n\n+ [InstructRL4Pix: Training Diffusion for Image Editing by Reinforcement Learning](https://arxiv.org/abs/2406.09973) (2024-06-14) \n[](https://bair.berkeley.edu/blog/2023/07/14/ddpo/)\n\n+ [Diffusion-RPO: Aligning Diffusion Models through Relative Preference Optimization](https://arxiv.org/abs/2406.06382) (2024-06-10)\n > Note: new evaluation metric: style alignment\n\n+ [Margin-aware Preference Optimization for Aligning Diffusion Models without Reference](https://arxiv.org/abs/2406.06424) (2024-06-10)\n\n+ [ReNO: Enhancing One-step Text-to-Image Models through Reward-based Noise Optimization](https://arxiv.org/abs/2406.04312) (2024-06-06) \n[](https://github.com/ExplainableML/ReNO)\n\n+ [Step-aware Preference Optimization: Aligning Preference with Denoising Performance at Each Step](https://arxiv.org/abs/2406.04314) (2024-06-06)\n\n+ [Improving GFlowNets for Text-to-Image Diffusion Alignment](https://arxiv.org/abs/2406.00633) (2024-06-02)\n > Note: Improves text-to-image alignment with reward function\n\n+ [Enhancing Reinforcement Learning Finetuned Text-to-Image Generative Model Using Reward Ensemble](https://link.springer.com/chapter/10.1007/978-3-031-63031-6_19) (2024-06-01)\n\n+ [Boost Your Own Human Image Generation Model via Direct Preference Optimization with AI Feedback](https://arxiv.org/abs/2405.20216) (2024-05-30)\n\n+ [T2V-Turbo: Breaking the Quality Bottleneck of Video Consistency Model with Mixed Reward Feedback](https://arxiv.org/abs/2405.18750) (2024-05-29) \n [](https://github.com/Ji4chenLi/t2v-turbo) [](https://t2v-turbo.github.io/)\n\n+ [Curriculum Direct Preference Optimization for Diffusion and Consistency Models](https://arxiv.org/abs/2405.13637) (2024-05-22) \n\n+ [Class-Conditional self-reward mechanism for improved Text-to-Image models](https://arxiv.org/abs/2405.13473) (2024-05-22) \n [](https://github.com/safouaneelg/SRT2I)\n\n+ [Understanding and Evaluating Human Preferences for AI Generated Images with Instruction Tuning](https://arxiv.org/abs/2405.07346) (2024-05-12)\n\n+ [Deep Reward Supervisions for Tuning Text-to-Image Diffusion Models](https://arxiv.org/abs/2405.00760) (2024-05-01) \n\n+ [ID-Aligner: Enhancing Identity-Preserving Text-to-Image Generation with Reward Feedback Learning](https://arxiv.org/abs/2404.15449) (2024-04-23) \n [](https://github.com/Weifeng-Chen/ID-Aligner) [](https://idaligner.github.io)\n\n+ [Hyper-SD: Trajectory Segmented Consistency Model for Efficient Image Synthesis](https://arxiv.org/abs/2404.13686) (2024-04-21)\n[](https://huggingface.co/ByteDance/Hyper-SD) [](https://hyper-sd.github.io/) [](https://huggingface.co/spaces/ByteDance/Hyper-SDXL-1Step-T2I)\n >Note: Human feedback learning to enhance model performance in low-steps regime\n\n\n+ [Prompt Optimizer of Text-to-Image Diffusion Models for Abstract Concept Understanding](https://arxiv.org/abs/2404.11589) (2024-04-17)\n\n+ [ControlNet++: Improving Conditional Controls with Efficient Consistency Feedback](https://arxiv.org/abs/2404.07987) (2024-04-11) \n\n+ [UniFL: Improve Stable Diffusion via Unified Feedback Learning](https://arxiv.org/abs/2404.05595) (2024-04-08) \n\n+ [YaART: Yet Another ART Rendering Technology](https://arxiv.org/abs/2404.05666) (2024-04-08)\n\n+ [ByteEdit: Boost, Comply and Accelerate Generative Image Editing](https://arxiv.org/abs/2404.04860) (2024-04-07) \n [](https://byte-edit.github.io/)\n > Note: ByteEdit, feedback learning framework for Generative Image Editing tasks\n\n+ [Aligning Diffusion Models by Optimizing Human Utility](https://arxiv.org/abs/2404.04465) (2024-04-06) \n\n+ [Dynamic Prompt Optimizing for Text-to-Image Generation](https://arxiv.org/abs/2404.04095) (2024-04-05) \n[](https://github.com/Mowenyii/PAE)\n\n+ [Pixel-wise RL on Diffusion Models: Reinforcement Learning from Rich Feedback](https://arxiv.org/abs/2404.04356) (2024-04-05) \n\n+ [CoMat: Aligning Text-to-Image Diffusion Model with Image-to-Text Concept Matching](https://arxiv.org/abs/2404.03653) (2024-04-04) \n [](https://github.com/CaraJ7/CoMat) [](https://caraj7.github.io/comat/)\n\n+ [VersaT2I: Improving Text-to-Image Models with Versatile Reward](https://arxiv.org/abs/2403.18493) (2024-03-27) \n\n\n+ [Improving Text-to-Image Consistency via Automatic Prompt Optimization](https://arxiv.org/abs/2403.17804) (2024-03-26) \n\n+ [RL for Consistency Models: Faster Reward Guided Text-to-Image Generation](https://arxiv.org/abs/2404.03673) (2024-03-25) \n [](https://rlcm.owenoertell.com)\n [](https://github.com/Owen-Oertell/rlcm)\n\n+ [AGFSync: Leveraging AI-Generated Feedback for Preference Optimization in Text-to-Image Generation](https://arxiv.org/abs/2403.13352) (2024-03-20) \n\n+ [Reward Guided Latent Consistency Distillation](https://arxiv.org/abs/2403.11027) (2024-03-16) \n [](https://github.com/Ji4chenLi/rg-lcd) [](https://rg-lcd.github.io/)\n\n+ [Optimizing Negative Prompts for Enhanced Aesthetics and Fidelity in Text-To-Image Generation](https://arxiv.org/abs/2403.07605) (2024-03-12)\n\n\n+ [Debiasing Text-to-Image Diffusion Models](https://arxiv.org/abs/2402.14577) (2024-02-22)\n\n\n\n+ [Universal Prompt Optimizer for Safe Text-to-Image Generation](https://arxiv.org/abs/2402.10882) (2024-02-16, NAACL 2024) \n [](https://github.com/wzongyu/POSI)\n\n+ [Social Reward: Evaluating and Enhancing Generative AI through Million-User Feedback from an Online Creative Community](https://arxiv.org/abs/2402.09872) (2024-02-15, ICLR 2024)\n [](https://github.com/Picsart-AI-Research/Social-Reward)\n\n+ [A Dense Reward View on Aligning Text-to-Image Diffusion with Preference](https://arxiv.org/abs/2402.08265) (2024-02-13, ICML 2024) \n [](https://github.com/Shentao-YANG/Dense_Reward_T2I)\n\n+ [Confronting Reward Overoptimization for Diffusion Models: A Perspective of Inductive and Primacy Biases](https://arxiv.org/abs/2402.08552) (2024-02-13, ICML 2024)\n[](https://github.com/ZiyiZhang27/tdpo)\n\n+ [PRDP: Proximal Reward Difference Prediction for Large-Scale Reward Finetuning of Diffusion Models](https://arxiv.org/abs/2402.08714) (2024-02-13)\n\n+ [Human Aesthetic Preference-Based Large Text-to-Image Model Personalization: Kandinsky Generation as an Example](https://arxiv.org/abs/2402.06389) (2024-02-09)\n\n+ [Divide and Conquer: Language Models can Plan and Self-Correct for Compositional Text-to-Image Generation](https://arxiv.org/abs/2401.15688) (2024-01-28)\n[](https://github.com/zhenyuw16/CompAgent_code)\n\n+ [Large-scale Reinforcement Learning for Diffusion Models](https://arxiv.org/abs/2401.12244) (2024-01-20)\n\n+ [Parrot: Pareto-optimal Multi-Reward Reinforcement Learning Framework for Text-to-Image Generation](https://arxiv.org/abs/2401.05675) (2024-01-11)\n\n+ [InstructVideo: Instructing Video Diffusion Models with Human Feedback](https://arxiv.org/abs/2312.12490) (2023-12-19) \n [](https://instructvideo.github.io)\n\n+ [Rich Human Feedback for Text-to-Image Generation](https://arxiv.org/abs/2312.10240) (2023-12-15, CVPR 2024) \n\n+ [iDesigner: A High-Resolution and Complex-Prompt Following Text-to-Image Diffusion Model for Interior Design](https://arxiv.org/abs/2312.04326) (2023-12-07)\n\n+ [InstructBooth: Instruction-following Personalized Text-to-Image Generation](https://arxiv.org/abs/2312.03011) (2023-12-04) [](https://sites.google.com/view/instructbooth)\n\n+ [DreamSync: Aligning Text-to-Image Generation with Image Understanding Feedback](https://arxiv.org/abs/2311.17946) (2023-11-29) \n\n+ [Enhancing Diffusion Models with Text-Encoder Reinforcement Learning](https://arxiv.org/abs/2311.15657) (2023-11-27)\n[](https://github.com/chaofengc/TexForce)\n\n+ [AdaDiff: Adaptive Step Selection for Fast Diffusion](https://arxiv.org/abs/2311.14768) (2023-11-24) \n\n+ [Using Human Feedback to Fine-tune Diffusion Models without Any Reward Model](https://arxiv.org/abs/2311.13231) (2023-11-22)\n[](https://github.com/yk7333/d3po)\n\n+ [Diffusion Model Alignment Using Direct Preference Optimization](https://arxiv.org/abs/2311.12908) (2023-11-21) \n [](https://github.com/SalesforceAIResearch/DiffusionDPO)[](https://github.com/huggingface/diffusers/tree/main/examples/research_projects/diffusion_dpo) [](https://blog.salesforceairesearch.com/diffusion-dpo/)\n\n+ [BeautifulPrompt: Towards Automatic Prompt Engineering for Text-to-Image Synthesis](https://arxiv.org/abs/2311.06752) (2023-11-12) \n\n\n+ [Quality Diversity through Human Feedback: Towards Open-Ended Diversity-Driven Optimization](https://arxiv.org/abs/2310.12103) (2023-10-18, ICML 2024)\n[](https://github.com/ld-ing/qdhf) [](https://liding.info/qdhf/)\n\n\n+ [Aligning Text-to-Image Diffusion Models with Reward Backpropagation](https://arxiv.org/abs/2310.03739) (2023-10-05) \n [](https://github.com/mihirp1998/AlignProp/)\n [](https://align-prop.github.io/)\n\n+ [Directly Fine-Tuning Diffusion Models on Differentiable Rewards](https://arxiv.org/abs/2309.17400) (2023-09-29)\n\n+ [LayoutLLM-T2I: Eliciting Layout Guidance from LLM for Text-to-Image Generation](https://arxiv.org/abs/2308.05095) (2023-08-09, ACM MM 2023) \n [](https://github.com/LayoutLLM-T2I/LayoutLLM-T2I) [](https://layoutllm-t2i.github.io/) [](https://huggingface.co/leigangqu/LayoutLLM-T2I/tree/main)\n\n\n+ [FABRIC: Personalizing Diffusion Models with Iterative Feedback](https://arxiv.org/abs/2307.10159) (2023-07-19) \n[](https://github.com/sd-fabric/fabric) [\n\n\n+ [Divide, Evaluate, and Refine: Evaluating and Improving Text-to-Image Alignment with Iterative VQA Feedback](https://arxiv.org/abs/2307.04749) (2023-07-10, NeurIPS 2023) \n [](https://github.com/1jsingh/Divide-Evaluate-and-Refine) [](https://1jsingh.github.io/divide-evaluate-and-refine)\n\n+ [Censored Sampling of Diffusion Models Using 3 Minutes of Human Feedback](https://arxiv.org/abs/2307.02770) (2023-07-06, NeurIPS 2023)\n[](https://github.com/tetrzim/diffusion-human-feedback)\n > Note: Censored generation using a reward model\n\n+ [StyleDrop: Text-to-Image Generation in Any Style](https://arxiv.org/abs/2306.00983) (2023-06-01)\n [](https://styledrop.github.io/)\n > Note: Iterative Training with Feedback\n\n\n+ [RealignDiff: Boosting Text-to-Image Diffusion Model with Coarse-to-fine Semantic Re-alignment](https://arxiv.org/abs/2305.19599) (2023-05-31)\n\n\n+ [DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Models](https://arxiv.org/abs/2305.16381) (2023-05-25, NeurIPS 2023) \n [](https://github.com/google-research/google-research/tree/master/dpok) [](https://sites.google.com/view/dpok-t2i-diffusion/home)\n\n+ [Training Diffusion Models with Reinforcement Learning](https://arxiv.org/abs/2305.13301) (2023-05-22) \n[](https://github.com/jannerm/ddpo)\n[Website](https://img.shields.io/badge/Website-9cf)](https://rl-diffusion.github.io/)\n\n+ [ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation](https://arxiv.org/abs/2304.05977) (2023-04-12) \n [](https://github.com/THUDM/ImageReward)\n\n+ [Confidence-aware Reward Optimization for Fine-tuning Text-to-Image Models](https://arxiv.org/abs/2404.01863) (2023-04-02, ICLR 2024) \n\n\n+ [Human Preference Score: Better Aligning Text-to-Image Models with Human Preference](https://arxiv.org/abs/2303.14420) (2023-03-25) \n [](https://github.com/tgxs002/align_sd)\n [](https://tgxs002.github.io/align_sd_web/)\n\n+ [HIVE: Harnessing Human Feedback for Instructional Visual Editing](https://arxiv.org/abs/2303.09618) (2023-03-16) \n](https://github.com/salesforce/HIVE)\n\n+ [Aligning Text-to-Image Models using Human Feedback](https://arxiv.org/abs/2302.12192) (2023-02-23)\n\n+ [Optimizing Prompts for Text-to-Image Generation](https://arxiv.org/abs/2212.09611) (2022-12-19, NeurIPS 2023) \n [](https://github.com/microsoft/LMOps/tree/main/promptist) \n [](https://huggingface.co/spaces/microsoft/Promptist)\n\n\n\n\n## 5. Quality Assessment for AIGC\n\n### 5.1. Image Quality Assessment for AIGC\n\n+ [Descriptive Image Quality Assessment in the Wild](https://arxiv.org/abs/2405.18842) (2024-05-29) \n [](https://depictqa.github.io/depictqa-wild/)\n\n+ [PKU-AIGIQA-4K: A Perceptual Quality Assessment Database for Both Text-to-Image and Image-to-Image AI-Generated Images](https://arxiv.org/abs/2404.18409) (2024-04-29)\n[](https://github.com/jiquan123/I2IQA)\n\n+ [Large Multi-modality Model Assisted AI-Generated Image Quality Assessment](https://arxiv.org/abs/2404.17762) (2024-04-27) \n [](https://github.com/wangpuyi/MA-AGIQA)\n\n+ [Adaptive Mixed-Scale Feature Fusion Network for Blind AI-Generated Image Quality Assessment](https://arxiv.org/abs/2404.15163) (2024-04-23) \n\n+ [PCQA: A Strong Baseline for AIGC Quality Assessment Based on Prompt Condition](https://arxiv.org/abs/2404.13299) (2024-04-20) \n\n+ [AIGIQA-20K: A Large Database for AI-Generated Image Quality Assessment](https://arxiv.org/abs/2404.03407) (2024-04-04) \n [](https://www.modelscope.cn/datasets/lcysyzxdxc/AIGCQA-30K-Image/summary)\n \n+ [AIGCOIQA2024: Perceptual Quality Assessment of AI Generated Omnidirectional Images](https://arxiv.org/abs/2404.01024) (2024-04-01)\n\n+ [Bringing Textual Prompt to AI-Generated Image Quality Assessment](https://arxiv.org/abs/2403.18714) (2024-03-27, ICME 2024) \n [](https://github.com/Coobiw/IP-IQA)\n\n+ [TIER: Text-Image Encoder-based Regression for AIGC Image Quality Assessment](https://arxiv.org/abs/2401.03854) (2024-01-08) \n[](https://github.com/jiquan123/TIER)\n\n+ [PSCR: Patches Sampling-based Contrastive Regression for AIGC Image Quality Assessment](https://arxiv.org/abs/2312.05897) (2023-12-10)\n[](https://github.com/jiquan123/PSCR)\n\n+ [Exploring the Naturalness of AI-Generated Images](https://arxiv.org/abs/2312.05476) (2023-12-09) \n [](https://github.com/zijianchen98/AGIN)\n \n+ [PKU-I2IQA: An Image-to-Image Quality Assessment Database for AI Generated Images](https://arxiv.org/abs/2311.15556) (2023-11-27) \n [](https://github.com/jiquan123/I2IQA)\n\n+ [Appeal and quality assessment for AI-generated images](https://ieeexplore.ieee.org/document/10178486) (2023-07-18)\n\n+ [AIGCIQA2023: A Large-scale Image Quality Assessment Database for AI Generated Images: from the Perspectives of Quality, Authenticity and Correspondence](https://arxiv.org/abs/2307.00211) (2023-07-01) \n \n+ [AGIQA-3K: An Open Database for AI-Generated Image Quality Assessment](https://arxiv.org/abs/2306.04717) (2023-06-07) \n [](https://github.com/lcysyzxdxc/AGIQA-3k-Database)\n\n+ [A Perceptual Quality Assessment Exploration for AIGC Images](https://arxiv.org/abs/2303.12618) (2023-03-22)\n\n+ [SPS: A Subjective Perception Score for Text-to-Image Synthesis](https://ieeexplore.ieee.org/abstract/document/9401705) (2021-04-27) \n\n\n+ [GIQA: Generated Image Quality Assessment](https://arxiv.org/abs/2003.08932) (2020-03-19)\n[](https://github.com/cientgu/GIQA)\n\n### 5.2. Aesthetic Predictors for Generated Images\n\n+ [Multi-modal Learnable Queries for Image Aesthetics Assessment](https://arxiv.org/abs/2405.01326) (2024-05-02, ICME 2024) \n\n+ Aesthetic Scorer extension for SD Automatic WebUI (2023-01-15) \n [](https://github.com/vladmandic/sd-extension-aesthetic-scorer)\n\n\n+ Simulacra Aesthetic-Models (2022-07-09) \n [](https://github.com/crowsonkb/simulacra-aesthetic-models)\n\n+ [Rethinking Image Aesthetics Assessment: Models, Datasets and Benchmarks](https://www.researchgate.net/publication/362037160_Rethinking_Image_Aesthetics_Assessment_Models_Datasets_and_Benchmarks) (2022-07-01)\n[](https://github.com/woshidandan/TANet-image-aesthetics-and-quality-assessment)\n\n\n+ LAION-Aesthetics_Predictor V2: CLIP+MLP Aesthetic Score Predictor (2022-06-26) \n [](https://github.com/christophschuhmann/improved-aesthetic-predictor)\n [](http://captions.christoph-schuhmann.de/aesthetic_viz_laion_sac+logos+ava1-l14-linearMSE-en-2.37B.html)\n [](https://laion.ai/blog/laion-aesthetics/#laion-aesthetics-v2)\n\n\n+ LAION-Aesthetics_Predictor V1 (2022-05-21) \n [](https://github.com/LAION-AI/aesthetic-predictor)\n [](https://laion.ai/blog/laion-aesthetics/#laion-aesthetics-v1)\n\n\n\n\n\n## 6. Study and Rethinking\n\n### 6.1. Evaluation of Evaluations\n+ [GAIA: Rethinking Action Quality Assessment for AI-Generated Videos](https://arxiv.org/abs/2406.06087) (2024-06-10)\n\n+ [Who Evaluates the Evaluations? Objectively Scoring Text-to-Image Prompt Coherence Metrics with T2IScoreScore (TS2)](https://arxiv.org/abs/2404.04251) (2024-04-05) \n [](https://github.com/michaelsaxon/T2IScoreScore)\n [](https://t2iscorescore.github.io) [](https://huggingface.co/datasets/saxon/T2IScoreScore)\n\n\n\n### 6.2. Survey\n\n+ [Multimodal Chain-of-Thought Reasoning: A Comprehensive Survey](https://arxiv.org/abs/2503.12605) (2025-03-23)\n\n+ [Survey of Bias In Text-to-Image Generation: Definition, Evaluation, and Mitigation](https://arxiv.org/abs/2404.01030) (2024-05-01)\n\n+ [Motion Generation: A Survey of Generative Approaches and Benchmarks](https://arxiv.org/abs/2507.05419) (2025-07-07)\n\n+ [Advancing Talking Head Generation: A Comprehensive Survey of Multi-Modal Methodologies, Datasets, Evaluation Metrics, and Loss Functions](https://arxiv.org/abs/2507.02900) (2025-06-23)\n\n+ [A Survey of Automatic Evaluation Methods on Text, Visual and Speech Generations](https://arxiv.org/abs/2506.10019) (2025-06-06)\n\n+ [Survey of Video Diffusion Models: Foundations, Implementations, and Applications](https://arxiv.org/abs/2504.16081) (2025-04-22)\n\n+ [A Survey on Quality Metrics for Text-to-Image Generation](https://arxiv.org/abs/2403.11821) (2024-03-18)\n\n+ [A Survey of AI-Generated Video Evaluation](https://arxiv.org/abs/2410.19884) (2024-10-24) \n\n+ [A Survey of Multimodal-Guided Image Editing with Text-to-Image Diffusion Models](https://arxiv.org/abs/2406.14555) (2024-06-20) \n\n+ [From Sora What We Can See: A Survey of Text-to-Video Generation](https://arxiv.org/abs/2405.10674) (2024-05-17) \n [](https://github.com/soraw-ai/Awesome-Text-to-Video-Generation) \n > Note: Refer to Section 3.4 for Evaluation Datasets and Metrics\n\n+ [A Survey on Personalized Content Synthesis with Diffusion Models](https://arxiv.org/abs/2405.05538) (2024-05-09) \n > Note: Refere to Section 6 for Evaluation Datasets and Metrics\n\n+ [A Survey on Long Video Generation: Challenges, Methods, and Prospects](https://arxiv.org/abs/2403.16407) (2024-03-25)\n\n > Note: Refer to table 2 for evaluation metrics for long video generation\n\n+ [Sora as an AGI World Model? A Complete Survey on Text-to-Video Generation](https://arxiv.org/abs/2403.05131) (2024-03-08) \n\n+ [State of the Art on Diffusion Models for Visual Computing](https://arxiv.org/abs/2310.07204) (2023-10-11) \n > Note: Refer to Section 9 for Metrics\n\n\n+ [AI-Generated Images as Data Source: The Dawn of Synthetic Era](https://arxiv.org/abs/2310.01830) (2023-10-03)\n[](https://github.com/mwxely/AIGS)\n >Note: Refer to Section 4.2 for Evaluation Metrics\n \n+ [A Survey on Video Diffusion Models](https://arxiv.org/abs/2310.10647) (2023-10-06) \n [](https://github.com/ChenHsing/Awesome-Video-Diffusion-Models)\n > Note: Refer to Section 2.3 for Evaluation Datasets and Metrics\n\n+ [Text-to-image Diffusion Models in Generative AI: A Survey](https://arxiv.org/abs/2303.07909) (2023-03-14) \n > Note: Refer to Section 5 for Evaulation from Techincal and Ethical Perspective\n\n+ [Image synthesis: a review of methods, datasets, evaluation metrics, and future outlook](https://link-springer-com.remotexs.ntu.edu.sg/article/10.1007/s10462-023-10434-2#Sec20) (2023-02-28)\n > Note: Refer to section 4 for evaluation metrics \n\n+ [Adversarial Text-to-Image Synthesis: A Review](https://arxiv.org/abs/2101.09983) (2021-01-25) \n > Note: Refer to Section 5 for Evaluation of T2I Models\n\n+ [Generative Adversarial Networks (GANs): An Overview of Theoretical Model, Evaluation Metrics, and Recent Developments](https://arxiv.org/abs/2005.13178) (2020-05-27)\n > Note: Refer to section 2.2 for Evaluation Metrics\n\n+ [What comprises a good talking-head video generation?: A Survey and Benchmark](https://arxiv.org/abs/2005.03201) (2020-05-07) \n[](https://github.com/lelechen63/talking-head-generation-survey)\n\n+ [A Survey and Taxonomy of Adversarial Neural Networks for Text-to-Image Synthesis](https://arxiv.org/abs/1910.09399) (2019-10-21) \n > Note: Refer to Section 5 for Benchmark and Evaluation\n\n+ [Recent Progress on Generative Adversarial Networks (GANs): A Survey](https://ieeexplore.ieee.org/document/8667290) (2019-03-14)\n > Note: Refer to section 5 for Evaluation Metrics\n\n+ [Video Description: A Survey of Methods, Datasets and Evaluation Metrics](https://arxiv.org/abs/1806.00186) (2018-06-01)\n > Note: Refer to section 5 for Evaluation Metrics\n\n### 6.3. Study\n\n+ [A-Bench: Are LMMs Masters at Evaluating AI-generated Images?](https://arxiv.org/abs/2406.03070) (2024-06-05)\n [](https://github.com/Q-Future/A-Bench)\n\n+ [On the Content Bias in Fréchet Video Distance](https://arxiv.org/abs/2404.12391) (2024-04-18, CVPR 2024) \n [](https://github.com/songweige/content-debiased-fvd)\n [](https://content-debiased-fvd.github.io)\n\n+ [Text-to-Image Synthesis With Generative Models: Methods, Datasets, Performance Metrics, Challenges, and Future Direction](https://ieeexplore.ieee.org/abstract/document/10431766) (2024-02-09)\n\n+ [On the Evaluation of Generative Models in Distributed Learning Tasks](https://arxiv.org/abs/2310.11714) (2023-10-18)\n\n+ [Recent Advances in Text-to-Image Synthesis: Approaches, Datasets and Future Research Prospects](https://ieeexplore.ieee.org/abstract/document/10224242) (2023-08-18)\n\n+ [Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models](https://arxiv.org/abs/2306.04675) (2023-06-07, NeurIPS 2023) \n [](https://github.com/layer6ai-labs/dgm-eval)\n\n+ [Toward Verifiable and Reproducible Human Evaluation for Text-to-Image Generation](https://arxiv.org/abs/2304.01816) (2023-04-04, CVPR 2023)\n\n+ [Revisiting the Evaluation of Image Synthesis with GANs](https://arxiv.org/abs/2304.01999) (2023-04-04)\n\n\n+ [A Study on the Evaluation of Generative Models](https://arxiv.org/abs/2206.10935) (2022-06-22)\n\n+ [REALY: Rethinking the Evaluation of 3D Face Reconstruction](https://arxiv.org/abs/2203.09729) (2022-03-18) \n[](https://github.com/czh-98/REALY)\n[](https://realy3dface.com/)\n\n+ [On Aliased Resizing and Surprising Subtleties in GAN Evaluation](https://arxiv.org/abs/2104.11222) (2021-04-22) \n [](https://github.com/GaParmar/clean-fid) \n [](https://www.cs.cmu.edu/~clean-fid/)\n\n+ [Pros and Cons of GAN Evaluation Measures: New Developments](https://arxiv.org/abs/2103.09396) (2021-03-17) \n\n+ [On the Robustness of Quality Measures for GANs](https://arxiv.org/abs/2201.13019) (2022-01-31, ECCV 2022) \n [](https://github.com/MotasemAlfarra/R-FID-Robustness-of-Quality-Measures-for-GANs)\n\n+ [Multimodal Image Synthesis and Editing: The Generative AI Era](https://arxiv.org/abs/2112.13592) (2021-12-27) \n[](https://github.com/fnzhan/Generative-AI)\n\n+ [An Analysis of Text-to-Image Synthesis](https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3852950) (2021-05-25)\n\n+ [Pros and Cons of GAN Evaluation Measures](https://arxiv.org/abs/1802.03446) (2018-02-09) \n\n+ [A Note on the Inception Score](https://arxiv.org/abs/1801.01973) (2018-01-06)\n\n+ [An empirical study on evaluation metrics of generative adversarial networks](https://arxiv.org/abs/1806.07755) (2018-06-19) \n [](https://github.com/xuqiantong/GAN-Metrics)\n\n+ [Are GANs Created Equal? A Large-Scale Study](https://arxiv.org/abs/1711.10337) (2017-11-28, NeurIPS 2018)\n\n+ [A note on the evaluation of generative models](https://arxiv.org/abs/1511.01844) (2015-11-05)\n\n+ [Appeal prediction for AI up-scaled Images](https://arxiv.org/abs/2502.14013) (2024-12-12) [](https://github.com/Telecommunication-Telemedia-Assessment/ai_upscaling)\n\n### 6.4. Competition\n+ [NTIRE 2024 Quality Assessment of AI-Generated Content Challenge](https://arxiv.org/abs/2404.16687) (2024-04-25)\n\n+ [CVPR 2023 Text Guided Video Editing Competition](https://arxiv.org/abs/2310.16003) (2023-10-24) \n[](https://github.com/showlab/loveu-tgve-2023)\n[](https://sites.google.com/view/loveucvpr23/track4)\n\n\n## 7. Other Useful Resources\n + Stanford Course: CS236 \"Deep Generative Models\" - Lecture 15 \"Evaluation of Generative Models\" [[slides]](https://deepgenerativemodels.github.io/assets/slides/lecture15.pdf)\n\n+ [Use of Neural Signals to Evaluate the Quality of Generative Adversarial Network Performance in Facial Image Generation](https://arxiv.org/abs/1811.04172) (2018-11-10)\n\n\n\n"
}
]