ICAFusion: Iterative Cross-Attention Guided Feature Fusion for Multispectral Object Detection
\n\n### Introduction\nIn this paper, we propose a novel feature fusion framework of dual cross-attention transformers to model global feature interaction and capture complementary information across modalities simultaneously. In addition, we introdece an iterative interaction mechanism into dual cross-attention transformers, which shares parameters among block-wise multimodal transformers to reduce model complexity and computation cost. The proposed method is general and effective to be integrated into different detection frameworks and used with different backbones. Experimental results on KAIST, FLIR, and VEDAI datasets show that the proposed method achieves superior performance and faster inference, making it suitable for various practical scenarios. \n\nPaper download in: https://arxiv.org/pdf/2308.07504.pdf\n\n### Overview\n\n ![](\"https://github.com/chanchanchan97/ICAFusion/assets/39607836/05a71809-0182-487d-9013-442497a996fd\")
\n
\n\n Fig 1. Overview of our multispectral object detection framework
\n\n ![](\"https://github.com/chanchanchan97/ICAFusion/assets/39607836/b82ba614-22da-421c-89e9-53d6d535ee36\")
\n
\n\n### Installation\nClone repo and install requirements.txt in a Python>=3.8.0 conda environment, including PyTorch>=1.12.\n```\ngit clone https://github.com/chanchanchan97/ICAFusion.git\ncd ICAFusion\npip install -r requirements.txt\n```\n\n### Datasets\n - **KAIST** \nLink：https://pan.baidu.com/s/1UdwQJH-cHVL91pkMW-ij6g \nCode：ig3y\n\n - **FLIR-aligned** \nLink：https://pan.baidu.com/s/1ljr8qJYdz-60Lj-iVEHBvg \nCode：uqzs\n\n - **VEDAI** \nLink：https://pan.baidu.com/s/1UKSI0Go0Ddt62tXNIySz9w\nCode：5ett\n\n\n### Weights\n - **KAIST** \nLink：https://pan.baidu.com/s/18UXctOSgjp6EUcJXIGbWTQ\nCode：9eku\n\n - **FLIR-aligned** \nWe fixed some bugs and now update the results on the FLIR dataset with the model weights as follows: \n~~Link：https://pan.baidu.com/s/1VZbsTE4o6bw2XBypPW3zoA\nCode：xli9~~ \nLink: https://pan.baidu.com/s/1fEFOEyzVIxNCMjJjuujJUQ?pwd=a81f \ncode: a81f\n\n### Results update\n - **FLIR**\n\n| Methods | person | car | bicycle | mAP@50 |\n|----------------------|----------|----------|----------|----------|\n| ~~yolov5_ICAFusion~~ | ~~81.6~~ | ~~89.0~~ | ~~66.9~~ | ~~79.2~~ |\n| yolov5_ICAFusion | 84.9 | 89.8 | 73.8 | 82.8 |\n\n - **M3FD**\n\n| People | Car | Bus | Motorcycle | Lamp | Truck | mAP@50 |\n|--------|------|------|------------|------|-------|--------|\n| 82.3 | 93.4 | 93.9 | 89.6 | 82.2 | 87.8 | 88.2 |\n\n- **DroneVehicle**\n\n| Car | Truck | Bus | Van | freight_car | mAP@50 |\n|------|-------|------|------|-------------|--------|\n| 98.0 | 84.2 | 97.1 | 71.0 | 73.8 | 84.8 |\n\n- **DVTOP**\n\n| Car | bicycle | person | mAP@50 |\n|------|------|-------------|--------|\n| 92.6 |83.6|84.6 |86.9|\n\n- **SeaDroneSee**\n\n| Swimmer | Floater | Boat | mAP@50 |\n|------|------|-------------|--------|\n|14.7 |69.5|99.5 |61.2|\n\n### Files\n**Note**: This is the txt files for evaluation. We continuously optimize our codes, which results in the difference in detection performance. However, the codes of module for multimodal feature fusion still remain consistent with the methods proposed in this paper.\n\n - **KAIST**\nLink：https://pan.baidu.com/s/1N7SNEPXKX7KFaO2Th7vq2g \nCode：zijw \n### Our new works \n- Multispectral State-Space Feature Fusion: Bridging Shared and Cross-Parametric Interactions for Object Detection **[[paper]](https://arxiv.org/abs/2507.14643)** **[[code]](https://github.com/61s61min/MS2Fusion)** \n- IRDFusion: Iterative Relation-Map Difference guided Feature Fusion for Multispectral Object Detection **[[paper]](https://arxiv.org/html/2509.09085v1)** **[[code]](https://github.com/61s61min/IRDFusion.git)**\n\n### Citation\nIf you find our work useful in your research, please consider citing:\n```\n@article{SHEN2023109913,\n title={ICAFusion: Iterative Cross-Attention Guided Feature Fusion for Multispectral Object Detection},\n author={Shen, Jifeng and Chen, Yifei and Liu, Yue and Zuo, Xin and Fan, Heng and Yang, Wankou},\n journal={Pattern Recognition},\n pages={109913},\n year={2023},\n issn={0031-3203},\n doi={https://doi.org/10.1016/j.patcog.2023.109913},\n author={Jifeng Shen and Yifei Chen and Yue Liu and Xin Zuo and Heng Fan and Wankou Yang},\n}\n```\n"
},
{
"path": "confluence.py",
"content": "\"\"\"\nAuthor: Andrew Shepley\nContact: asheple2@une.edu.au\nSource: Confluence\nMethods\na) assign_boxes_to_classes\nb) normalise_coordinates\nc) confluence_nms - returns maxima scoring box, removes false positives using confluence - efficient\nd) confluence - returns most confluent box, removes false positives using confluence - less efficient but better box\n\"\"\"\n\nfrom collections import defaultdict\nimport numpy as np\n\ndef assign_boxes_to_classes(bounding_boxes, classes, scores):\n \"\"\"\n Parameters: \n bounding_boxes: list of bounding boxes (x1,y1,x2,y2)\n classes: list of class identifiers (int value, e.g. 1 = person)\n scores: list of class confidence scores (0.0-1.0)\n Returns:\n boxes_to_classes: defaultdict(list) containing mapping to bounding boxes and confidence scores to class\n \"\"\"\n boxes_to_classes = defaultdict(list)\n for each_box, each_class, each_score in zip(bounding_boxes, classes, scores):\n if each_score >= 0.05:\n boxes_to_classes[each_class].append(np.array([each_box[0],each_box[1],each_box[2],each_box[3], each_score]))\n return boxes_to_classes\n\ndef normalise_coordinates(x1, y1, x2, y2,min_x,max_x,min_y,max_y):\n \"\"\"\n Parameters: \n x1, y1, x2, y2: bounding box coordinates to normalise\n min_x,max_x,min_y,max_y: minimum and maximum bounding box values (min = 0, max = 1)\n Returns:\n Normalised bounding box coordinates (scaled between 0 and 1)\n \"\"\"\n x1, y1, x2, y2 = (x1-min_x)/(max_x-min_x), (y1-min_y)/(max_y-min_y), (x2-min_x)/(max_x-min_x), (y2-min_y)/(max_y-min_y)\n return x1, y1, x2, y2\n\ndef confluence_nms(bounding_boxes,scores,classes,confluence_thr,gaussian,score_thr=0.05,sigma=0.5): \n \"\"\"\n Parameters:\n bounding_boxes: list of bounding boxes (x1,y1,x2,y2)\n classes: list of class identifiers (int value, e.g. 1 = person)\n scores: list of class confidence scores (0.0-1.0)\n confluence_thr: value between 0 and 2, with optimum from 0.5-0.8\n gaussian: boolean switch to turn gaussian decaying of suboptimal bounding box confidence scores (setting to False results in suppression of suboptimal boxes)\n score_thr: class confidence score\n sigma: used in gaussian decaying. A smaller value causes harsher decaying.\n Returns:\n output: A dictionary mapping class identity to final retained boxes (and corresponding confidence scores)\n \"\"\"\n \n class_mapping = assign_boxes_to_classes(bounding_boxes, classes, scores)\n output = {}\n for each_class in class_mapping:\n dets = np.array(class_mapping[each_class])\n retain = []\n while dets.size > 0:\n max_idx = np.argmax(dets[:, 4], axis=0)\n dets[[0, max_idx], :] = dets[[max_idx, 0], :]\n retain.append(dets[0, :])\n x1, y1, x2, y2 = dets[0, 0], dets[0, 1], dets[0, 2], dets[0, 3]\n \n min_x = np.minimum(x1, dets[1:, 0])\n min_y = np.minimum(y1, dets[1:, 1])\n max_x = np.maximum(x2, dets[1:, 2]) \n max_y = np.maximum(y2, dets[1:, 3])\n \n x1, y1, x2, y2 = normalise_coordinates(x1, y1, x2, y2,min_x,max_x,min_y,max_y)\n xx1, yy1, xx2, yy2 = normalise_coordinates(dets[1:, 0], dets[1:, 1], dets[1:, 2], dets[1:, 3],min_x,max_x,min_y,max_y)\n\n md_x1,md_x2,md_y1,md_y2 = abs(x1-xx1),abs(x2-xx2),abs(y1-yy1),abs(y2-yy2) \n manhattan_distance = (md_x1+md_x2+md_y1+md_y2)\n\n weights = np.ones_like(manhattan_distance)\n\n if (gaussian == True):\n gaussian_weights = np.exp(-((1-manhattan_distance) * (1-manhattan_distance)) / sigma)\n weights[manhattan_distance<=confluence_thr]=gaussian_weights[manhattan_distance<=confluence_thr]\n else:\n weights[manhattan_distance<=confluence_thr]=manhattan_distance[manhattan_distance<=confluence_thr]\n\n dets[1:, 4] *= weights\n to_reprocess = np.where(dets[1:, 4] >= score_thr)[0]\n dets = dets[to_reprocess + 1, :] \n output[each_class]=retain\n\n return output\n\ndef confluence(bounding_boxes,scores,classes,confluence_thr,gaussian,score_thr=0.05,sigma=0.5):\n \"\"\"\n Parameters:\n bounding_boxes: list of bounding boxes (x1,y1,x2,y2)\n classes: list of class identifiers (int value, e.g. 1 = person)\n scores: list of class confidence scores (0.0-1.0)\n confluence_thr: value between 0 and 2, with optimum from 0.5-0.8\n gaussian: boolean switch to turn gaussian decaying of suboptimal bounding box confidence scores (setting to False results in suppression of suboptimal boxes)\n score_thr: class confidence score\n sigma: used in gaussian decaying. A smaller value causes harsher decaying.\n Returns:\n output: A dictionary mapping class identity to final retained boxes (and corresponding confidence scores)\n \"\"\"\n\n class_mapping = assign_boxes_to_classes(bounding_boxes, classes, scores)\n output = {}\n for each_class in class_mapping:\n dets = np.array(class_mapping[each_class])\n retain = []\n while dets.size > 0:\n confluence_scores,proximities = [],[]\n while len(confluence_scores) Fig 2. Illustration of the proposed DMFF module
\n