Repository: LutingWang/awesome-knowledge-distillation-for-object-detection
Branch: main
Commit: eab179583b4f
Files: 8
Total size: 36.3 KB

Directory structure:
gitextract_90f2ms82/

├── .github/
│   ├── CODE_OF_CONDUCT.md
│   ├── CONTRIBUTING.md
│   └── workflows/
│       └── lint.yaml
├── .markdownlint-cli2.yaml
├── .tool-versions
├── .vscode/
│   └── settings.json
├── LICENSE
└── README.md

================================================
FILE CONTENTS
================================================

================================================
FILE: .github/CODE_OF_CONDUCT.md
================================================
# Contributor Covenant Code of Conduct

## Our Pledge

We as members, contributors, and leaders pledge to make participation in our
community a harassment-free experience for everyone, regardless of age, body
size, visible or invisible disability, ethnicity, sex characteristics, gender
identity and expression, level of experience, education, socio-economic status,
nationality, personal appearance, race, caste, color, religion, or sexual
identity and orientation.

We pledge to act and interact in ways that contribute to an open, welcoming,
diverse, inclusive, and healthy community.

## Our Standards

Examples of behavior that contributes to a positive environment for our
community include:

* Demonstrating empathy and kindness toward other people
* Being respectful of differing opinions, viewpoints, and experiences
* Giving and gracefully accepting constructive feedback
* Accepting responsibility and apologizing to those affected by our mistakes,
  and learning from the experience
* Focusing on what is best not just for us as individuals, but for the overall
  community

Examples of unacceptable behavior include:

* The use of sexualized language or imagery, and sexual attention or advances of
  any kind
* Trolling, insulting or derogatory comments, and personal or political attacks
* Public or private harassment
* Publishing others' private information, such as a physical or email address,
  without their explicit permission
* Other conduct which could reasonably be considered inappropriate in a
  professional setting

## Enforcement Responsibilities

Community leaders are responsible for clarifying and enforcing our standards of
acceptable behavior and will take appropriate and fair corrective action in
response to any behavior that they deem inappropriate, threatening, offensive,
or harmful.

Community leaders have the right and responsibility to remove, edit, or reject
comments, commits, code, wiki edits, issues, and other contributions that are
not aligned to this Code of Conduct, and will communicate reasons for moderation
decisions when appropriate.

## Scope

This Code of Conduct applies within all community spaces, and also applies when
an individual is officially representing the community in public spaces.
Examples of representing our community include using an official e-mail address,
posting via an official social media account, or acting as an appointed
representative at an online or offline event.

## Enforcement

Instances of abusive, harassing, or otherwise unacceptable behavior may be
reported to the community leaders responsible for enforcement at
[wangluting@buaa.edu.cn](mailto:wangluting@buaa.edu.cn).
All complaints will be reviewed and investigated promptly and fairly.

All community leaders are obligated to respect the privacy and security of the
reporter of any incident.

## Enforcement Guidelines

Community leaders will follow these Community Impact Guidelines in determining
the consequences for any action they deem in violation of this Code of Conduct:

### 1. Correction

**Community Impact**: Use of inappropriate language or other behavior deemed
unprofessional or unwelcome in the community.

**Consequence**: A private, written warning from community leaders, providing
clarity around the nature of the violation and an explanation of why the
behavior was inappropriate. A public apology may be requested.

### 2. Warning

**Community Impact**: A violation through a single incident or series of
actions.

**Consequence**: A warning with consequences for continued behavior. No
interaction with the people involved, including unsolicited interaction with
those enforcing the Code of Conduct, for a specified period of time. This
includes avoiding interactions in community spaces as well as external channels
like social media. Violating these terms may lead to a temporary or permanent
ban.

### 3. Temporary Ban

**Community Impact**: A serious violation of community standards, including
sustained inappropriate behavior.

**Consequence**: A temporary ban from any sort of interaction or public
communication with the community for a specified period of time. No public or
private interaction with the people involved, including unsolicited interaction
with those enforcing the Code of Conduct, is allowed during this period.
Violating these terms may lead to a permanent ban.

### 4. Permanent Ban

**Community Impact**: Demonstrating a pattern of violation of community
standards, including sustained inappropriate behavior, harassment of an
individual, or aggression toward or disparagement of classes of individuals.

**Consequence**: A permanent ban from any sort of public interaction within the
community.

## Attribution

This Code of Conduct is adapted from the [Contributor Covenant][homepage],
version 2.1, available at
[https://www.contributor-covenant.org/version/2/1/code_of_conduct.html][v2.1].

Community Impact Guidelines were inspired by
[Mozilla's code of conduct enforcement ladder][Mozilla CoC].

For answers to common questions about this code of conduct, see the FAQ at
[https://www.contributor-covenant.org/faq][FAQ]. Translations are available at
[https://www.contributor-covenant.org/translations][translations].

[homepage]: https://www.contributor-covenant.org
[v2.1]: https://www.contributor-covenant.org/version/2/1/code_of_conduct.html
[Mozilla CoC]: https://github.com/mozilla/diversity
[FAQ]: https://www.contributor-covenant.org/faq
[translations]: https://www.contributor-covenant.org/translations


================================================
FILE: .github/CONTRIBUTING.md
================================================
# Contribution Guidelines

You’re more than welcome to contribute to the project!
To keep things organized and smooth, please follow these guidelines.

## Awesomeness

Awesome works in the field of detection KD are those with a strong influence.
That said, the field is still under rapid development, so newly published works with high potential are also acceptable, for example works from top conferences like CVPR.
Before adding a new work to the list, the awesomeness should be discussed in the pull request.

## Formatting

Each work should be formatted as below:

```md
MethodName. *ConfName YYYY*.
\[[source](URL)\]
<[repo](URL)>
\- Description

- Paper Title
- FirstName LastName and FirstName LastName
```

The rendered output should look like:

> KD. *NeurIPS 2015*.
> \[[arXiv](http://arxiv.org/abs/1503.02531)\]
> \- Description
>
> - Distilling the Knowledge in a Neural Network
> - Geoffrey Hinton and Oriol Vinyals and Jeff Dean

If the work does not have a `MethodName`, the field can be omitted.
A work can have multiple `source` fields.

## More Information

For more information, please refer to the [Contribution Guidelines of Awesome](https://github.com/sindresorhus/awesome/blob/main/contributing.md).


================================================
FILE: .github/workflows/lint.yaml
================================================
name: lint
on:
  pull_request:
  push:
    branches:
      - main

jobs:
  lint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - uses: DavidAnson/markdownlint-cli2-action@v11


================================================
FILE: .markdownlint-cli2.yaml
================================================
config:
  MD013: false


================================================
FILE: .tool-versions
================================================
nodejs 16.5.0


================================================
FILE: .vscode/settings.json
================================================
{
    "cSpell.words": [
        "DRKD",
        "DSIG",
        "FGFI",
        "SSIM",
        "TADF",
        "Xplore"
    ]
}


================================================
FILE: LICENSE
================================================
Creative Commons Legal Code

CC0 1.0 Universal

    CREATIVE COMMONS CORPORATION IS NOT A LAW FIRM AND DOES NOT PROVIDE
    LEGAL SERVICES. DISTRIBUTION OF THIS DOCUMENT DOES NOT CREATE AN
    ATTORNEY-CLIENT RELATIONSHIP. CREATIVE COMMONS PROVIDES THIS
    INFORMATION ON AN "AS-IS" BASIS. CREATIVE COMMONS MAKES NO WARRANTIES
    REGARDING THE USE OF THIS DOCUMENT OR THE INFORMATION OR WORKS
    PROVIDED HEREUNDER, AND DISCLAIMS LIABILITY FOR DAMAGES RESULTING FROM
    THE USE OF THIS DOCUMENT OR THE INFORMATION OR WORKS PROVIDED
    HEREUNDER.

Statement of Purpose

The laws of most jurisdictions throughout the world automatically confer
exclusive Copyright and Related Rights (defined below) upon the creator
and subsequent owner(s) (each and all, an "owner") of an original work of
authorship and/or a database (each, a "Work").

Certain owners wish to permanently relinquish those rights to a Work for
the purpose of contributing to a commons of creative, cultural and
scientific works ("Commons") that the public can reliably and without fear
of later claims of infringement build upon, modify, incorporate in other
works, reuse and redistribute as freely as possible in any form whatsoever
and for any purposes, including without limitation commercial purposes.
These owners may contribute to the Commons to promote the ideal of a free
culture and the further production of creative, cultural and scientific
works, or to gain reputation or greater distribution for their Work in
part through the use and efforts of others.

For these and/or other purposes and motivations, and without any
expectation of additional consideration or compensation, the person
associating CC0 with a Work (the "Affirmer"), to the extent that he or she
is an owner of Copyright and Related Rights in the Work, voluntarily
elects to apply CC0 to the Work and publicly distribute the Work under its
terms, with knowledge of his or her Copyright and Related Rights in the
Work and the meaning and intended legal effect of CC0 on those rights.

1. Copyright and Related Rights. A Work made available under CC0 may be
protected by copyright and related or neighboring rights ("Copyright and
Related Rights"). Copyright and Related Rights include, but are not
limited to, the following:

  i. the right to reproduce, adapt, distribute, perform, display,
     communicate, and translate a Work;
 ii. moral rights retained by the original author(s) and/or performer(s);
iii. publicity and privacy rights pertaining to a person's image or
     likeness depicted in a Work;
 iv. rights protecting against unfair competition in regards to a Work,
     subject to the limitations in paragraph 4(a), below;
  v. rights protecting the extraction, dissemination, use and reuse of data
     in a Work;
 vi. database rights (such as those arising under Directive 96/9/EC of the
     European Parliament and of the Council of 11 March 1996 on the legal
     protection of databases, and under any national implementation
     thereof, including any amended or successor version of such
     directive); and
vii. other similar, equivalent or corresponding rights throughout the
     world based on applicable law or treaty, and any national
     implementations thereof.

2. Waiver. To the greatest extent permitted by, but not in contravention
of, applicable law, Affirmer hereby overtly, fully, permanently,
irrevocably and unconditionally waives, abandons, and surrenders all of
Affirmer's Copyright and Related Rights and associated claims and causes
of action, whether now known or unknown (including existing as well as
future claims and causes of action), in the Work (i) in all territories
worldwide, (ii) for the maximum duration provided by applicable law or
treaty (including future time extensions), (iii) in any current or future
medium and for any number of copies, and (iv) for any purpose whatsoever,
including without limitation commercial, advertising or promotional
purposes (the "Waiver"). Affirmer makes the Waiver for the benefit of each
member of the public at large and to the detriment of Affirmer's heirs and
successors, fully intending that such Waiver shall not be subject to
revocation, rescission, cancellation, termination, or any other legal or
equitable action to disrupt the quiet enjoyment of the Work by the public
as contemplated by Affirmer's express Statement of Purpose.

3. Public License Fallback. Should any part of the Waiver for any reason
be judged legally invalid or ineffective under applicable law, then the
Waiver shall be preserved to the maximum extent permitted taking into
account Affirmer's express Statement of Purpose. In addition, to the
extent the Waiver is so judged Affirmer hereby grants to each affected
person a royalty-free, non transferable, non sublicensable, non exclusive,
irrevocable and unconditional license to exercise Affirmer's Copyright and
Related Rights in the Work (i) in all territories worldwide, (ii) for the
maximum duration provided by applicable law or treaty (including future
time extensions), (iii) in any current or future medium and for any number
of copies, and (iv) for any purpose whatsoever, including without
limitation commercial, advertising or promotional purposes (the
"License"). The License shall be deemed effective as of the date CC0 was
applied by Affirmer to the Work. Should any part of the License for any
reason be judged legally invalid or ineffective under applicable law, such
partial invalidity or ineffectiveness shall not invalidate the remainder
of the License, and in such case Affirmer hereby affirms that he or she
will not (i) exercise any of his or her remaining Copyright and Related
Rights in the Work or (ii) assert any associated claims and causes of
action with respect to the Work, in either case contrary to Affirmer's
express Statement of Purpose.

4. Limitations and Disclaimers.

 a. No trademark or patent rights held by Affirmer are waived, abandoned,
    surrendered, licensed or otherwise affected by this document.
 b. Affirmer offers the Work as-is and makes no representations or
    warranties of any kind concerning the Work, express, implied,
    statutory or otherwise, including without limitation warranties of
    title, merchantability, fitness for a particular purpose, non
    infringement, or the absence of latent or other defects, accuracy, or
    the present or absence of errors, whether or not discoverable, all to
    the greatest extent permissible under applicable law.
 c. Affirmer disclaims responsibility for clearing rights of other persons
    that may apply to the Work or any use thereof, including without
    limitation any person's Copyright and Related Rights in the Work.
    Further, Affirmer disclaims responsibility for obtaining any necessary
    consents, permissions or other rights required for any use of the
    Work.
 d. Affirmer understands and acknowledges that Creative Commons is not a
    party to this document and has no duty or obligation with respect to
    this CC0 or use of the Work.


================================================
FILE: README.md
================================================
# Awesome Knowledge Distillation for Object Detection [![Awesome](https://awesome.re/badge.svg)](https://awesome.re)

[![lint](https://github.com/LutingWang/awesome-knowledge-distillation-for-object-detection/actions/workflows/lint.yaml/badge.svg)](https://github.com/LutingWang/awesome-knowledge-distillation-for-object-detection/actions/workflows/lint.yaml)

A curated list of **awesome** distillation techniques designed for object detectors.

Parameters compression and accuracy boosting are core problems for object detection towards practical application, where knowledge distillation (KD) is one of the most popular solutions.
KD aims at training the compact model (student) by transferring knowledge from a high-capacity model (teacher).
Papers and codes are listed.

## Contents

- [Knowledge Distillation for General Object Detectors](#knowledge-distillation-for-general-object-detectors)
  - [Feature Distillation](#feature-distillation)
    - [Foreground Masks](#foreground-masks)
      - [Ground Truth Guided](#ground-truth-guided)
      - [Prediction Guided](#prediction-guided)
      - [Attention Guided](#attention-guided)
      - [Miscellaneous Foreground Masks](#miscellaneous-foreground-masks)
    - [Miscellaneous Feature Distillation](#miscellaneous-feature-distillation)
  - [Instance Distillation](#instance-distillation)
  - [Label Assignment Distillation](#label-assignment-distillation)
  - [Balancing between Tasks](#balancing-between-tasks)
  - [Miscellaneous Knowledge Distillation for General Object Detectors](#miscellaneous-knowledge-distillation-for-general-object-detectors)
- [Knowledge Distillation for Specific Object Detectors](#knowledge-distillation-for-specific-object-detectors)
  - [Knowledge Distillation for GFL](#knowledge-distillation-for-gfl)
  - [Knowledge Distillation for DETR](#knowledge-distillation-for-detr)
- [Knowledge Distillation for Heterogeneous Object Detectors](#knowledge-distillation-for-heterogeneous-object-detector-pairs)
- [Multi Teacher Knowledge Distillation for Object Detectors](#multi-teacher-knowledge-distillation-for-object-detectors)
- [Teacher Free Knowledge Distillation for Object Detectors](#teacher-free-knowledge-distillation-for-object-detectors)
- [Miscellaneous](#miscellaneous)
- [Newly Published Papers](#newly-published-papers)

## Knowledge Distillation for General Object Detectors

*NeurIPS 2017*.
\[[NeurIPS](https://proceedings.neurips.cc/paper/2017/hash/e1e32e235eee1f970470a3a6658dfdd5-Abstract.html)\]
\- A new framework to learn compact and fast object detection networks with improved accuracy using knowledge distillation and hint learning.

- Learning Efficient Object Detection Models with Knowledge Distillation
- Guobin Chen and Wongun Choi and Xiang Yu and Tony Han and Manmohan Chandraker

Mimic. *CVPR 2017*.
\[[CVF](http://openaccess.thecvf.com/content_cvpr_2017/html/Li_Mimicking_Very_Efficient_CVPR_2017_paper.html)\]
\[[IEEE Xplore](http://ieeexplore.ieee.org/abstract/document/8100259/)\]
\- A fully convolutional feature mimic framework to train very efficient CNN based detectors, which do not need ImageNet pre-training and achieve competitive performance as the large and slow models.

- Mimicking Very Efficient Network for Object Detection
- Quanquan Li and Shengying Jin and Junjie Yan

### Feature Distillation

#### Foreground Masks

##### Ground Truth Guided

FGFI. *CVPR 2019*.
\[[CVF](http://openaccess.thecvf.com/content_CVPR_2019/html/Wang_Distilling_Object_Detectors_With_Fine-Grained_Feature_Imitation_CVPR_2019_paper.html)\]
\[[IEEE Xplore](https://ieeexplore.ieee.org/abstract/document/8953432/)\]
\[[arXiv](http://arxiv.org/abs/1906.03609)\]
<[GitHub](https://github.com/twangnh/Distilling-Object-Detectors)>
\- A fine-grained feature imitation method exploiting the cross-location discrepancy of feature response.

- Distilling Object Detectors With Fine-Grained Feature Imitation
- Tao Wang and Li Yuan and Xiaopeng Zhang and Jiashi Feng

DeFeat. *CVPR 2021*.
\[[CVF](http://openaccess.thecvf.com/content/CVPR2021/html/Guo_Distilling_Object_Detectors_via_Decoupled_Features_CVPR_2021_paper.html)\]
\[[IEEE Xplore](https://ieeexplore.ieee.org/abstract/document/9578919/)\]
\[[arXiv](http://arxiv.org/abs/2103.14475)\]
\- A novel distillation algorithm via decoupled features for learning a better student detector.

- Distilling Object Detectors via Decoupled Features
- Jianyuan Guo and Kai Han and Yunhe Wang and Han Wu and Xinghao Chen and Chunjing Xu and Chang Xu

##### Prediction Guided

FRS. *NeurIPS 2021*.
\[[NeurIPS](https://proceedings.neurips.cc/paper_files/paper/2021/hash/29c0c0ee223856f336d7ea8052057753-Abstract.html)\]
\[[OpenReview](https://openreview.net/forum?id=_bOfK2k_7R)\]
\[[arXiv](http://arxiv.org/abs/2111.00674)\]
\- A novel Feature-Richness Score (FRS) method to choose important features that improve generalized detectability during distilling.

- Distilling Object Detectors with Feature Richness
- Zhixing Du and Rui Zhang and Ming Chang and Xishan Zhang and Shaoli Liu and Tianshi Chen and Yunji Chen

PGD. *ECCV 2022*.
\[[ECVA](https://www.ecva.net/papers/eccv_2022/papers_ECCV/papers/136690123.pdf)\]
\[[Springer](https://link.springer.com/chapter/10.1007/978-3-031-20077-9_8)\]
\[[arXiv](http://arxiv.org/abs/2203.05469)\]
<[GitHub](https://github.com/ChenhongyiYang/PGD)>
\- Distill on the key predictive regions of the teacher.

- Prediction-Guided Distillation for Dense Object Detection
- Chenhongyi Yang and Mateusz Ochal and Amos Storkey and Elliot J Crowley

TBD. *PR*.
\[[ScienceDirect](https://www.sciencedirect.com/science/article/abs/pii/S0031320323000213)\]
\[[arXiv](https://arxiv.org/abs/2208.03006)\]
\- Alleviates the misalignment between classification score and localization quality via Harmony Score and Task-Balanced Distillation.

- Task-balanced distillation for object detection
- Ruining Tang and Zhenyu Liu and Yangguang Li and Yiguo Song and Hui Liu and Qide Wang and Jing Shao and Guifang Duan and Jianrong Tan

##### Attention Guided

FKD. *ICLR 2021*.
\[[OpenReview](https://openreview.net/forum?id=uKhGRvM8QNH)\]
<[GitHub](https://github.com/ArchipLab-LinfengZhang/Object-Detection-Knowledge-Distillation-ICLR2021)>
\- Attention-guided distillation and non-local distillation.

- Improve Object Detection with Feature-based Knowledge Distillation: Towards Accurate and Efficient Detectors
- Chunting Zhou and Graham Neubig and Jiatao Gu

FKD. *TPAMI*.
\[[IEEE Xplore](https://ieeexplore.ieee.org/abstract/document/10198386/)\]
\- A structured knowledge distillation scheme, including attention-guided distillation and non-local distillation.

- Structured Knowledge Distillation for Accurate and Efficient Object Detection
- Linfeng Zhang and Kaisheng Ma

FGD. *CVPR 2022*.
\[[CVF](https://openaccess.thecvf.com/content/CVPR2022/html/Yang_Focal_and_Global_Knowledge_Distillation_for_Detectors_CVPR_2022_paper.html)\]
\[[IEEE Xplore](https://ieeexplore.ieee.org/abstract/document/9879869/)\]
\[[arXiv](http://arxiv.org/abs/2111.11837)\]
<[GitHub](https://github.com/yzd-v/FGD)>
\- Focal distillation separates the fore-ground and background, while global distillation rebuilds the relation between different pixels and transfers it from teachers to students.

- Focal and Global Knowledge Distillation for Detectors
- Zhendong Yang and Zhe Li and Xiaohu Jiang and Yuan Gong and Zehuan Yuan and Danpei Zhao and Chun Yuan

GLAMD. *ECCV 2022*.
\[[ECVA](https://www.ecva.net/papers/eccv_2022/papers_ECCV/papers/136700456.pdf)\]
\[[Springer](https://link.springer.com/chapter/10.1007/978-3-031-20080-9_27)\]
\- Divide the feature maps into several patches and apply an attention mechanism for both the entire feature area and each patch.

- GLAMD: Global and Local Attention Mask Distillation for Object Detectors
- Younho Jang and Wheemyung Shin and Jinbeom Kim and Simon Woo and Sung-Ho Bae

##### Miscellaneous Foreground Masks

CD. *ICCV 2021*.
\[[CVF](http://openaccess.thecvf.com/content/ICCV2021/html/Shu_Channel-Wise_Knowledge_Distillation_for_Dense_Prediction_ICCV_2021_paper.html)\]
\[[IEEE Xplore](https://ieeexplore.ieee.org/abstract/document/9710762/)\]
\[[arXiv](http://arxiv.org/abs/2011.13256)\]
<[GitHub](https://github.com/irfanICMLL/TorchDistiller/tree/main/SemSeg-distill)>
\- Normalize the activation map of each channel to obtain a soft probability map.

- Channel-wise Knowledge Distillation for Dense Prediction
- Changyong Shu and Yifan Liu and Jianfei Gao and Zheng Yan and Chunhua Shen

#### Miscellaneous Feature Distillation

DRKD. *IJCAI 2023*.
\[[arXiv](https://arxiv.org/abs/2302.05637)\]
\- Dual relation knowledge distillation, including pixel-wise relation distillation and instance-wise relation distillation

- Dual Relation Knowledge Distillation for Object Detection
- Zhenliang Ni and Fukui Yang and Shengzhao Wen and Gang Zhang

### Instance Distillation

GID. *CVPR 2021*.
\[[CVF](http://openaccess.thecvf.com/content/CVPR2021/html/Dai_General_Instance_Distillation_for_Object_Detection_CVPR_2021_paper.html)\]
\[[IEEE Xplore](https://ieeexplore.ieee.org/abstract/document/9577671/)\]
\[[arXiv](http://arxiv.org/abs/2103.02340)\]
\- A novel distillation method for detection tasks based on discriminative instances without considering the positive or negative distinguished by GT.

- General Instance Distillation for Object Detection
- Xing Dai and Zeren Jiang and Zhao Wu and Yiping Bao and Zhicheng Wang and Si Liu and Erjin Zhou

DSIG. *ICCV 2021*.
\[[CVF](http://openaccess.thecvf.com/content/ICCV2021/html/Chen_Deep_Structured_Instance_Graph_for_Distilling_Object_Detectors_ICCV_2021_paper.html)\]
\[[IEEE Xplore](https://ieeexplore.ieee.org/abstract/document/9711100/)\]
\[[arXiv](http://arxiv.org/abs/2109.12862)\]
<[GitHub](https://github.com/dvlab-research/Dsig)>
\- A simple knowledge structure to exploit and encode information inside the detection system to  facilitate detector knowledge distillation.

- Deep Structured Instance Graph for Distilling Object Detectors
- Yixin Chen and Pengguang Chen and Shu Liu and Liwei Wang and Jiaya Jia

ICD. *NeurIPS 2021*.
\[[NeurIPS](https://proceedings.neurips.cc/paper_files/paper/2021/hash/892c91e0a653ba19df81a90f89d99bcd-Abstract.html)\]
\[[OpenReview](https://openreview.net/forum?id=k7aeAz4Vbb)\]
\[[arXiv](http://arxiv.org/abs/2110.12724)\]
<[GitHub](https://github.com/MegEngine/ICD)>
\- An instance-conditional distillation framework to find desired knowledge.

- Instance-Conditional Knowledge Distillation for Object Detection
- Zijian Kang and Peizhen Zhang and Xiangyu Zhang and Jian Sun and Nanning Zheng

### Label Assignment Distillation

LAD. *WACV 2022*.
\[[CVF](https://openaccess.thecvf.com/content/WACV2022/html/Nguyen_Improving_Object_Detection_by_Label_Assignment_Distillation_WACV_2022_paper.html)\]
\[[IEEE Xplore](https://ieeexplore.ieee.org/abstract/document/9706993/)\]
\[[arXiv](http://arxiv.org/abs/2108.10520)\]
<[MMDet](https://github.com/open-mmlab/mmdetection/tree/master/configs/lad)>
\- Use the teacher network to generate labels for the student, through the hard labels dynamically  assigned by the teacher.

- Improving Object Detection by Label Assignment Distillation
- Chuong H. Nguyen and Thuy C. Nguyen and Tuan N. Tang and Nam L. H. Phan

### Balancing between Tasks

TADF.
\[[arXiv](http://arxiv.org/abs/2006.13108)\]
\- A general distillation framework that adaptively transfers knowledge from teacher to student  according to the task specific prior.

- Distilling Object Detectors with Task Adaptive Regularization
- Ruoyu Sun and Fuhui Tang and Xiaopeng Zhang and Hongkai Xiong and Qi Tian

BCKD. *ICCV 2023*
\[[CVF](https://openaccess.thecvf.com/content/ICCV2023/html/Yang_Bridging_Cross-task_Protocol_Inconsistency_for_Distillation_in_Dense_Object_Detection_ICCV_2023_paper.html)\]
\[[IEEE Xplore](https://ieeexplore.ieee.org/abstract/document/10377607)\]
\[[arXiv](http://arxiv.org/abs/2308.14286)\]
<[GitHub](https://github.com/TinyTigerPan/BCKD)>
\- A novel distillation method with cross-task consistent protocols, tailored for the dense object detection.

- Bridging Cross-task Protocol Inconsistency for Distillation in Dense Object Detection
- Longrong Yang and Xianpan Zhou and Xuewei Li and Liang Qiao and Zheyang Li and Ziwei Yang and Gaoang Wang and Xi Li

### Miscellaneous Knowledge Distillation for General Object Detectors

*AAAI 2022*.
\[[AAAI](https://ojs.aaai.org/index.php/AAAI/article/view/20018)\]
\[[arXiv](http://arxiv.org/abs/2112.04840)\]
\- RM takes the rank of candidate boxes from teachers as a new form of knowledge to distill. PFI attempts to correlate feature differences with prediction differences.

- Knowledge Distillation for Object Detection via Rank Mimicking and Prediction-Guided Feature Imitation
- Gang Li and Xiang Li and Yujie Wang and Shanshan Zhang and Yichao Wu and Ding Liang

*NeurIPS 2022*.
\[[OpenReview](https://openreview.net/forum?id=O3My0RK9s_R)\]
\[[arXiv](https://arxiv.org/abs/2211.13133v1)\]
<[GitHub](https://github.com/kornia/kornia)>
\- By taking into account additional contrast and structural cues, feature importance, correlation, and spatial dependence in the feature space are considered in the loss formulation.

- Structural Knowledge Distillation for Object Detection
- Philip De Rijk and Lukas Schneider and Marius Cordts and Dariu M Gavrila

CrossKD. CVPR 2024.
\[[CVF](https://openaccess.thecvf.com/content/CVPR2024/html/Wang_CrossKD_Cross-Head_Knowledge_Distillation_for_Object_Detection_CVPR_2024_paper.html)\]
\[[IEEE Xplore](https://ieeexplore.ieee.org/document/10654891)\]
\[[arXiv](https://arxiv.org/abs/2306.11369)\]
<[GitHub](https://github.com/jbwang1997/CrossKD)>
\- Delivers the intermediate features of the student's detection head to the teacher's detection head

- CrossKD: Cross-Head Knowledge Distillation for Dense Object Detection
- Jiabao Wang and Yuming Chen and Zhaohui Zheng and Xiang Li and Ming-Ming Cheng and Qibin Hou

## Knowledge Distillation for Specific Object Detectors

### Knowledge Distillation for GFL

LD. *CVPR 2022*.
\[[CVF](https://openaccess.thecvf.com/content/CVPR2022/html/Zheng_Localization_Distillation_for_Dense_Object_Detection_CVPR_2022_paper.html)\]
\[[IEEE Xplore](https://ieeexplore.ieee.org/abstract/document/9878414/)\]
\[[arXiv](http://arxiv.org/abs/2102.12252)\]
<[GitHub](https://github.com/HikariTJU/LD)>
<[MMDet](https://github.com/open-mmlab/mmdetection/tree/master/configs/ld)>
\- Standard KD by adopting the general localization representation of bounding box.

- Localization Distillation for Dense Object Detection
- Zhaohui Zheng and Rongguang Ye and Ping Wang and Jun Wang and Dongwei Ren and Wangmeng Zuo

### Knowledge Distillation for DETR

DETRDistill. *ICCV 2023*.
\[[CVF](https://openaccess.thecvf.com/content/ICCV2023/html/Chang_DETRDistill_A_Universal_Knowledge_Distillation_Framework_for_DETR-families_ICCV_2023_paper.html)\]
\[[arXiv](http://arxiv.org/abs/2211.10156)\]
\- A novel knowledge distillation dedicated to DETR-families.

- DETRDistill: A Universal Knowledge Distillation Framework for DETR-families
- Jiahao Chang and Shuo Wang and Guangkai Xu and Zehui Chen and Chenhongyi Yang and Feng Zhao

D^3^ETR.
\[[arXiv](http://arxiv.org/abs/2211.09768)\]
\- Distills knowledge in decoder predictions and attention maps from the teachers to students.

- D^3^ETR: Decoder Distillation for Detection Transformer
- Xiaokang Chen and Jiahui Chen and Yan Liu and Gang Zeng

KD-DETR.
\[[arXiv](http://arxiv.org/abs/2211.08071)\]
\- A general knowledge distillation paradigm for DETR with consistent distillation points sampling.

- Knowledge Distillation for Detection Transformer with Consistent Distillation Points Sampling
- Yu Wang and Xin Li and Shengzhao Wen and Fukui Yang and Wanping Zhang and Gang Zhang and Haocheng Feng and Junyu Han and Errui Ding

## Knowledge Distillation for Heterogeneous Object Detector Pairs

G-DetKD. *ICCV 2021*.
\[[CVF](http://openaccess.thecvf.com/content/ICCV2021/html/Yao_G-DetKD_Towards_General_Distillation_Framework_for_Object_Detectors_via_Contrastive_ICCV_2021_paper.html)\]
\[[IEEE Xplore](https://ieeexplore.ieee.org/abstract/document/9711293/)\]
\[[arXiv](http://arxiv.org/abs/2108.07482)\]
\- A novel semantic-guided feature imitation technique, which automatically performs soft matching between feature pairs across all pyramid levels to provide the optimal guidance to the student.

- G-DetKD: Towards General Distillation Framework for Object Detectors via Contrastive and Semantic-guided Feature Imitation
- Lewei Yao and Renjie Pi and Hang Xu and Wei Zhang and Zhenguo Li and Tong Zhang

HEAD. *ECCV 2022*.
\[[ECVA](https://www.ecva.net/papers/eccv_2022/papers_ECCV/papers/136690310.pdf)\]
\[[Springer](https://link.springer.com/chapter/10.1007/978-3-031-20077-9_19)\]
\[[arXiv](https://arxiv.org/abs/2207.05345)\]
<[GitHub](https://github.com/LutingWang/HEAD)>
\- HEtero-Assists Distillation leveraging heterogeneous detection heads as assistants to guide the optimization of the student detector.

- HEAD: HEtero-Assists Distillation for Heterogeneous Object Detectors
- Luting Wang and Xiaojie Li and Yue Liao and Zeren Jiang and Jianlong Wu and Fei Wang and Chen Qian and Si Liu

PKD. *NeurIPS 2022*.
\[[OpenReview](https://openreview.net/forum?id=Q9dj3MzY1o7)\]
\[[arXiv](https://arxiv.org/abs/2207.02039v2)\]
<[GitHub](https://github.com/open-mmlab/mmrazor)>
\- Imitate features with Pearson Correlation Coefficient to focus on the relational information from the teacher and relax constraints on the magnitude of the features.

- PKD: General Distillation Framework for Object Detectors via Pearson Correlation Coefficient
- Weihan Cao and Yifan Zhang and Jianfei Gao and Anda Cheng and Ke Cheng and Jian Cheng

UniKD. ICCV 2023.
\[[CVF](https://openaccess.thecvf.com/content/ICCV2023/html/Lao_UniKD_Universal_Knowledge_Distillation_for_Mimicking_Homogeneous_or_Heterogeneous_Object_ICCV_2023_paper.html)\]
\[[IEEE Xplore](https://ieeexplore.ieee.org/document/10376876)\]
\- Proposes a universal knowledge distillation framework that extracts detection-relevant knowledge embeddings via Adaptive Knowledge Extractor (AKE), enabling efficient knowledge transfer between homogeneous/heterogeneous detectors.

- UniKD: Universal Knowledge Distillation for Mimicking Homogeneous or Heterogeneous Object Detectors
- Shanshan Lao and Guanglu Song and Boxiao Liu and Yu Liu and Yujiu Yang

DetKDS. ICML 2024.
\[[ICML](https://icml.cc/virtual/2024/poster/34018)\]
<[GitHub](https://github.com/lliai/DetKDS)>
\- The first distillation search framework for object detection, automatically adapting to homogeneous/heterogeneous teacher-student pairs by searching for optimal distillation strategies to enhance model generalization.

- DetKDS: Knowledge Distillation Search for Object Detectors
- Lujun Li and Yufan Bao and Peijie Dong and Chuanguang Yang and Anggeng Li and Wenhan Luo and Qifeng Liu and Wei Xue and Yike Guo

## Multi Teacher Knowledge Distillation for Object Detectors

LLOD. ICML 2023.
\[[ICML](https://icml.cc/virtual/2023/poster/23867)\]
\[[arXiv](https://arxiv.org/abs/2308.09105)\]
<[GitHub](https://github.com/Shengcao-Cao/MTPD)>
\- Proposes a multi-teacher progressive distillation strategy, sequentially distilling knowledge from multiple teachers to bridge model capacity gaps, especially effective for knowledge transfer between heterogeneous architectures.

- Learning Lightweight Object Detectors via Multi-Teacher Progressive Distillation
- Shengcao Cao and Mengtian Li and James Hays and Deva Ramanan and Yu-Xiong Wang and Liangyan Gui

## Teacher Free Knowledge Distillation for Object Detectors

MimicDet. *ECCV 2020*.
\[[ECVA](https://www.ecva.net/papers/eccv_2020/papers_ECCV/papers/123590528.pdf)\]
\[[Springer](https://link.springer.com/chapter/10.1007/978-3-030-58568-6_32)\]
\[[arXiv](http://arxiv.org/abs/2009.11528)\]
\- A novel and efficient framework to train a one-stage detector by directly mimic the two-stage features, aiming to bridge the accuracy gap between one-stage and two-stage detector.

- MimicDet: Bridging the Gap Between One-Stage and Two-Stage Object Detection
- Xin Lu and Quanquan Li and Buyu Li and Junjie Yan

LabelEnc. *ECCV 2020*.
\[[ECVA](https://www.ecva.net/papers/eccv_2020/papers_ECCV/papers/123700528.pdf)\]
\[[Springer](https://link.springer.com/chapter/10.1007/978-3-030-58595-2_32)\]
\[[arXiv](http://arxiv.org/abs/2007.03282)\]
<[GitHub](https://github.com/megvii-model/LabelEnc)>
\- A new intermediate supervision method to boost the training of object detection systems.

- LabelEnc: A New Intermediate Supervision Method for Object Detection
- Miao Hao and Yitao Liu and Xiangyu Zhang and Jian Sun

HEAD. *ECCV 2022*.
\[[ECVA](https://www.ecva.net/papers/eccv_2022/papers_ECCV/papers/136690310.pdf)\]
\[[Springer](https://link.springer.com/chapter/10.1007/978-3-031-20077-9_19)\]
\[[arXiv](https://arxiv.org/abs/2207.05345)\]
<[GitHub](https://github.com/LutingWang/HEAD)>
\- HEtero-Assists Distillation leveraging heterogeneous detection heads as assistants to guide the optimization of the student detector.

- HEAD: HEtero-Assists Distillation for Heterogeneous Object Detectors
- Luting Wang and Xiaojie Li and Yue Liao and Zeren Jiang and Jianlong Wu and Fei Wang and Chen Qian and Si Liu

LGD. *AAAI 2022*.
\[[AAAI](https://ojs.aaai.org/index.php/AAAI/article/view/20240)\]
\[[arXiv](http://arxiv.org/abs/2109.11496)\]
\- The first self-distillation framework for general object detection.

- LGD: Label-Guided Self-Distillation for Object Detection
- Peizhen Zhang and Zijian Kang and Tong Yang and Xiangyu Zhang and Nanning Zheng and Jian Sun

SSD-Det. *ICCV 2023*.
\[[CVF](https://openaccess.thecvf.com/content/ICCV2023/html/Wu_Spatial_Self-Distillation_for_Object_Detection_with_Inaccurate_Bounding_Boxes_ICCV_2023_paper.html)\]
\[[IEEE Xplore](https://ieeexplore.ieee.org/abstract/document/10377611)\]
\[[arXiv](http://arxiv.org/abs/2307.12101)\]
\- Mine spatial information to refine the inaccurate box in a self-distillation fashion.

- Spatial Self-Distillation for Object Detection with Inaccurate Bounding Boxes
- Di Wu and Pengfei Chen and Xuehui Yu and Guorong Li and Zhenjun Han and Jianbin Jiao

## Miscellaneous

*TPAMI*.
\[[IEEE Xplore](https://ieeexplore.ieee.org/abstract/document/10070820)\]
\- A comprehensive survey of KD-based OD models.

- When Object Detection Meets Knowledge Distillation: A Survey
- Zhihui Li and Pengfei Xu and Xiaojun Chang and Luyao Yang and Yuanyuan Zhang and Lina Yao and Xiaojiang Chen

ScaleKD. *CVPR 2023*.
\[[CVF](https://openaccess.thecvf.com/content/CVPR2023/html/Zhu_ScaleKD_Distilling_Scale-Aware_Knowledge_in_Small_Object_Detector_CVPR_2023_paper.html)\]
\- Consists of a Scale-Decoupled Feature distillation module and a Cross-Scale Assistant.

- ScaleKD: Distilling Scale-Aware Knowledge in Small Object Detector
- Yichen Zhu and Qiqi Zhou and Ning Liu and Zhiyuan Xu and Zhicai Ou and Xiaofeng Mou and Jian Tang

## Newly Published Papers