[ { "path": "Metric/Metric_torch.py", "content": "import numpy as np\r\nfrom scipy.signal import convolve2d\r\nfrom Qabf import get_Qabf\r\nfrom Nabf import get_Nabf\r\nimport math\r\nimport torch\r\nimport torch.nn.functional as F\r\nimport torch.fft\r\nfrom ssim import ssim, ms_ssim\r\nfrom sklearn.metrics import normalized_mutual_info_score\r\n\r\ndef EN_function(image_tensor):\r\n histogram = torch.histc(image_tensor, bins=256, min=0, max=255)\r\n histogram = histogram / histogram.sum()\r\n entropy = -torch.sum(histogram * torch.log2(histogram + 1e-7))\r\n return entropy\r\n\r\ndef CE_function(ir_img_tensor, vi_img_tensor, f_img_tensor):\r\n ir_img_tensor = torch.sigmoid(ir_img_tensor)\r\n vi_img_tensor = torch.sigmoid(vi_img_tensor)\r\n f_img_tensor = torch.sigmoid(f_img_tensor)\r\n epsilon = 1e-7\r\n f_img_tensor = torch.clamp(f_img_tensor, epsilon, 1.0 - epsilon)\r\n true_tensor = (ir_img_tensor + vi_img_tensor) / 2\r\n true_tensor = torch.clamp(true_tensor, epsilon, 1.0 - epsilon)\r\n CE = F.binary_cross_entropy(f_img_tensor, true_tensor)\r\n return CE\r\n\r\ndef QNCIE_function(ir_img_tensor, vi_img_tensor, f_img_tensor):\r\n def normalize1(img_tensor):\r\n img_min = img_tensor.min()\r\n img_max = img_tensor.max()\r\n return (img_tensor - img_min) / (img_max - img_min)\r\n def NCC(img1, img2):\r\n mean1 = torch.mean(img1)\r\n mean2 = torch.mean(img2)\r\n numerator = torch.sum((img1 - mean1) * (img2 - mean2))\r\n denominator = torch.sqrt(torch.sum((img1 - mean1) ** 2) * torch.sum((img2 - mean2) ** 2))\r\n return numerator / (denominator + 1e-10)\r\n\r\n ir_img_tensor = normalize1(ir_img_tensor)\r\n vi_img_tensor = normalize1(vi_img_tensor)\r\n f_img_tensor = normalize1(f_img_tensor)\r\n\r\n NCCxy = NCC(ir_img_tensor, vi_img_tensor)\r\n NCCxf = NCC(ir_img_tensor, f_img_tensor)\r\n NCCyf = NCC(vi_img_tensor, f_img_tensor)\r\n R = torch.tensor([[1, NCCxy, NCCxf],\r\n [NCCxy, 1, NCCyf],\r\n [NCCxf, NCCyf, 1]], dtype=torch.float32)\r\n\r\n r = torch.linalg.eigvals(R).real\r\n K = 3\r\n b = 256\r\n HR = torch.sum(r * torch.log2(r / K) / K)\r\n HR = -HR / np.log2(b)\r\n QNCIE = 1 - HR.item()\r\n return QNCIE\r\n\r\n\r\ndef TE_function(ir_img_tensor, vi_img_tensor, f_img_tensor, q=1, ksize=256):\r\n def compute_entropy(img_tensor, q, ksize):\r\n img_tensor = img_tensor.view(-1).float()\r\n histogram = torch.histc(img_tensor, bins=ksize, min=0, max=ksize - 1)\r\n probabilities = histogram / torch.sum(histogram)\r\n if q == 1:\r\n entropy = -torch.sum(probabilities * torch.log2(probabilities + 1e-10))\r\n else:\r\n entropy = (1 / (q - 1)) * (1 - torch.sum(probabilities ** q))\r\n return entropy.item()\r\n\r\n TE_ir = compute_entropy(ir_img_tensor, q, ksize)\r\n TE_vi = compute_entropy(vi_img_tensor, q, ksize)\r\n TE_f = compute_entropy(f_img_tensor, q, ksize)\r\n TE = TE_ir + TE_vi - TE_f\r\n return TE\r\n\r\n\r\ndef EI_function(f_img_tensor):\r\n sobel_kernel_x = torch.tensor([[-1., 0., 1.],\r\n [-2., 0., 2.],\r\n [-1., 0., 1.]]).to(f_img_tensor.device)\r\n\r\n sobel_kernel_y = torch.tensor([[-1., -2., -1.],\r\n [0., 0., 0.],\r\n [1., 2., 1.]]).to(f_img_tensor.device)\r\n\r\n sobel_kernel_x = sobel_kernel_x.view(1, 1, 3, 3)\r\n sobel_kernel_y = sobel_kernel_y.view(1, 1, 3, 3)\r\n gx = F.conv2d(f_img_tensor.unsqueeze(0).unsqueeze(0), sobel_kernel_x, padding=1)\r\n gy = F.conv2d(f_img_tensor.unsqueeze(0).unsqueeze(0), sobel_kernel_y, padding=1)\r\n\r\n g = torch.sqrt(gx ** 2 + gy ** 2)\r\n EI = torch.mean(g).item()\r\n\r\n return EI\r\n\r\n\r\ndef SF_function(image_tensor):\r\n\r\n RF = image_tensor[1:, :] - image_tensor[:-1, :]\r\n CF = image_tensor[:, 1:] - image_tensor[:, :-1]\r\n RF1 = torch.sqrt(torch.mean(RF ** 2))\r\n CF1 = torch.sqrt(torch.mean(CF ** 2))\r\n\r\n SF = torch.sqrt(RF1 ** 2 + CF1 ** 2)\r\n return SF\r\n\r\n\r\ndef SD_function(image_tensor):\r\n m, n = image_tensor.shape\r\n u = torch.mean(image_tensor)\r\n SD = torch.sqrt(torch.sum((image_tensor - u) ** 2) / (m * n))\r\n return SD\r\n\r\ndef PSNR_function(A, B, F):\r\n A = A.float() / 255.0\r\n B = B.float() / 255.0\r\n F = F.float() / 255.0\r\n\r\n m, n = F.shape\r\n MSE_AF = torch.mean((F - A) ** 2)\r\n MSE_BF = torch.mean((F - B) ** 2)\r\n\r\n MSE = 0.5 * MSE_AF + 0.5 * MSE_BF\r\n PSNR = 20 * torch.log10(1 / torch.sqrt(MSE))\r\n\r\n return PSNR\r\n\r\n\r\ndef MSE_function(A, B, F):\r\n A = A.float() / 255.0\r\n B = B.float() / 255.0\r\n F = F.float() / 255.0\r\n\r\n m, n = F.shape\r\n MSE_AF = torch.mean((F - A) ** 2)\r\n MSE_BF = torch.mean((F - B) ** 2)\r\n\r\n MSE = 0.5 * MSE_AF + 0.5 * MSE_BF\r\n return MSE\r\n\r\ndef fspecial_gaussian(shape, sigma):\r\n m, n = [(ss-1.)/2. for ss in shape]\r\n y, x = np.ogrid[-m:m+1, -n:n+1]\r\n h = np.exp(-(x*x + y*y) / (2.*sigma*sigma))\r\n h[h < np.finfo(h.dtype).eps*h.max()] = 0\r\n sumh = h.sum()\r\n if sumh != 0:\r\n h /= sumh\r\n return h\r\n\r\ndef fspecial_gaussian(size, sigma):\r\n x = torch.linspace(-size[0]//2, size[0]//2, size[0])\r\n y = torch.linspace(-size[1]//2, size[1]//2, size[1])\r\n x, y = torch.meshgrid(x, y)\r\n g = torch.exp(-(x**2 + y**2) / (2 * sigma**2))\r\n return g / g.sum()\r\n\r\ndef convolve2d(input, kernel):\r\n kernel = kernel.unsqueeze(0).unsqueeze(0).to(input.device) # Add batch and channel dimensions\r\n return F.conv2d(input.unsqueeze(0).unsqueeze(0), kernel, padding=kernel.shape[2] // 2)[0][0]\r\n\r\ndef vifp_mscale(ref, dist):\r\n sigma_nsq = 2\r\n num = 0\r\n den = 0\r\n for scale in range(1, 5):\r\n N = 2 ** (4 - scale + 1) + 1\r\n win = fspecial_gaussian((N, N), N / 5)\r\n\r\n if scale > 1:\r\n ref = convolve2d(ref, win)\r\n dist = convolve2d(dist, win)\r\n ref = ref[::2, ::2]\r\n dist = dist[::2, ::2]\r\n\r\n mu1 = convolve2d(ref, win)\r\n mu2 = convolve2d(dist, win)\r\n mu1_sq = mu1 * mu1\r\n mu2_sq = mu2 * mu2\r\n mu1_mu2 = mu1 * mu2\r\n sigma1_sq = convolve2d(ref * ref, win) - mu1_sq\r\n sigma2_sq = convolve2d(dist * dist, win) - mu2_sq\r\n sigma12 = convolve2d(ref * dist, win) - mu1_mu2\r\n sigma1_sq[sigma1_sq < 0] = 0\r\n sigma2_sq[sigma2_sq < 0] = 0\r\n\r\n g = sigma12 / (sigma1_sq + 1e-10)\r\n sv_sq = sigma2_sq - g * sigma12\r\n\r\n g[sigma1_sq < 1e-10] = 0\r\n sv_sq[sigma1_sq < 1e-10] = sigma2_sq[sigma1_sq < 1e-10]\r\n sigma1_sq[sigma1_sq < 1e-10] = 0\r\n\r\n g[sigma2_sq < 1e-10] = 0\r\n sv_sq[sigma2_sq < 1e-10] = 0\r\n\r\n sv_sq[g < 0] = sigma2_sq[g < 0]\r\n g[g < 0] = 0\r\n sv_sq[sv_sq <= 1e-10] = 1e-10\r\n\r\n num += torch.sum(torch.log10(1 + g**2 * sigma1_sq / (sv_sq + sigma_nsq)))\r\n den += torch.sum(torch.log10(1 + sigma1_sq / sigma_nsq))\r\n\r\n vifp = num / den\r\n return vifp\r\n\r\ndef VIF_function(A, B, F):\r\n VIF = vifp_mscale(A, F) + vifp_mscale(B, F)\r\n return VIF\r\n\r\n\r\ndef CC_function(A, B, F):\r\n rAF = torch.sum((A - torch.mean(A)) * (F - torch.mean(F))) / torch.sqrt(torch.sum((A - torch.mean(A)) ** 2) * torch.sum((F - torch.mean(F)) ** 2))\r\n rBF = torch.sum((B - torch.mean(B)) * (F - torch.mean(F))) / torch.sqrt(torch.sum((B - torch.mean(B)) ** 2) * torch.sum((F - torch.mean(F)) ** 2))\r\n CC = torch.mean(torch.tensor([rAF, rBF]))\r\n return CC\r\n\r\ndef corr2(a, b):\r\n a = a - torch.mean(a)\r\n b = b - torch.mean(b)\r\n r = torch.sum(a * b) / torch.sqrt(torch.sum(a * a) * torch.sum(b * b))\r\n return r\r\n\r\ndef SCD_function(A, B, F):\r\n r = corr2(F - B, A) + corr2(F - A, B)\r\n return r\r\n\r\ndef Qabf_function(A, B, F):\r\n return get_Qabf(A, B, F)\r\n\r\ndef Nabf_function(A, B, F):\r\n return Nabf_function(A, B, F)\r\n\r\n\r\ndef Hab(im1, im2, gray_level):\r\n\thang, lie = im1.shape\r\n\tcount = hang * lie\r\n\tN = gray_level\r\n\th = np.zeros((N, N))\r\n\tfor i in range(hang):\r\n\t\tfor j in range(lie):\r\n\t\t\th[im1[i, j], im2[i, j]] = h[im1[i, j], im2[i, j]] + 1\r\n\th = h / np.sum(h)\r\n\tim1_marg = np.sum(h, axis=0)\r\n\tim2_marg = np.sum(h, axis=1)\r\n\tH_x = 0\r\n\tH_y = 0\r\n\tfor i in range(N):\r\n\t\tif (im1_marg[i] != 0):\r\n\t\t\tH_x = H_x + im1_marg[i] * math.log2(im1_marg[i])\r\n\tfor i in range(N):\r\n\t\tif (im2_marg[i] != 0):\r\n\t\t\tH_x = H_x + im2_marg[i] * math.log2(im2_marg[i])\r\n\tH_xy = 0\r\n\tfor i in range(N):\r\n\t\tfor j in range(N):\r\n\t\t\tif (h[i, j] != 0):\r\n\t\t\t\tH_xy = H_xy + h[i, j] * math.log2(h[i, j])\r\n\tMI = H_xy - H_x - H_y\r\n\treturn MI\r\n\r\ndef MI_function(A, B, F, gray_level=256):\r\n\tMIA = Hab(A, F, gray_level)\r\n\tMIB = Hab(B, F, gray_level)\r\n\tMI_results = MIA + MIB\r\n\treturn MI_results\r\n\r\ndef entropy(im, gray_level=256):\r\n\r\n hist, _ = np.histogram(im, bins=gray_level, range=(0, gray_level), density=True)\r\n H = -np.sum(hist * np.log2(hist + 1e-10))\r\n return H\r\n\r\ndef NMI_function(A, B, F, gray_level=256):\r\n\r\n MIA = Hab(A, F, gray_level)\r\n MIB = Hab(B, F, gray_level)\r\n MI_results = MIA + MIB\r\n\r\n H_A = entropy(A, gray_level)\r\n H_B = entropy(B, gray_level)\r\n\r\n NMI = 2 * MI_results / (H_A + H_B + 1e-10)\r\n\r\n return NMI\r\n\r\ndef AG_function(image_tensor):\r\n grady, gradx = torch.gradient(image_tensor)\r\n s = torch.sqrt((gradx ** 2 + grady ** 2) / 2)\r\n AG = torch.sum(s) / (image_tensor.shape[0] * image_tensor.shape[1])\r\n return AG\r\n\r\ndef SSIM_function(A, B, F):\r\n ssim_A = ssim(A, F)\r\n ssim_B = ssim(B, F)\r\n SSIM = (ssim_A + 1 * ssim_B) / 2\r\n return SSIM.item()\r\n\r\ndef MS_SSIM_function(A, B, F):\r\n ssim_A = ms_ssim(A, F)\r\n ssim_B = ms_ssim(B, F)\r\n MS_SSIM = (ssim_A + 1 * ssim_B) / 2\r\n return MS_SSIM.item()\r\n\r\ndef Nabf_function(A, B, F):\r\n Nabf = get_Nabf(A, B, F)\r\n return Nabf\r\n\r\ndef Qy_function(ir_img_tensor, vi_img_tensor, f_img_tensor):\r\n def gaussian_filter(window_size, sigma):\r\n gauss = torch.tensor([np.exp(-(x - window_size // 2) ** 2 / float(2 * sigma ** 2)) for x in range(window_size)], device=ir_img_tensor.device)\r\n gauss = gauss / gauss.sum()\r\n gauss = gauss.view(1, 1, -1).repeat(1, 1, 1, 1)\r\n return gauss\r\n\r\n def ssim_yang(img1, img2):\r\n window_size = 7\r\n sigma = 1.5\r\n window = gaussian_filter(window_size, sigma)\r\n window = window.expand(1, 1, window_size, window_size)\r\n\r\n mu1 = F.conv2d(img1, window, stride=1, padding=window_size // 2)\r\n mu2 = F.conv2d(img2, window, stride=1, padding=window_size // 2)\r\n mu1_sq = mu1.pow(2)\r\n mu2_sq = mu2.pow(2)\r\n mu1_mu2 = mu1 * mu2\r\n sigma1_sq = F.conv2d(img1.pow(2), window, padding=window_size // 2) - mu1_sq\r\n sigma2_sq = F.conv2d(img2.pow(2), window, padding=window_size // 2) - mu2_sq\r\n sigma12 = F.conv2d(img1 * img2, window, padding=window_size // 2) - mu1_mu2\r\n C1 = 0.01**2\r\n C2 = 0.03**2\r\n ssim_map = ((2 * mu1_mu2 + C1) * (2 * sigma12 + C2)) / ((mu1_sq + mu2_sq + C1) * (sigma1_sq + sigma2_sq + C2))\r\n mssim = ssim_map.mean().item()\r\n\r\n return mssim, ssim_map, sigma1_sq, sigma2_sq\r\n\r\n ir_img_tensor = ir_img_tensor.unsqueeze(0).unsqueeze(0).double()\r\n vi_img_tensor = vi_img_tensor.unsqueeze(0).unsqueeze(0).double()\r\n f_img_tensor = f_img_tensor.unsqueeze(0).unsqueeze(0).double()\r\n\r\n _, ssim_map1, sigma1_sq1, sigma2_sq1 = ssim_yang(ir_img_tensor, vi_img_tensor)\r\n _, ssim_map2, _, _ = ssim_yang(ir_img_tensor, f_img_tensor)\r\n _, ssim_map3, _, _ = ssim_yang(vi_img_tensor, f_img_tensor)\r\n bin_map = (ssim_map1 >= 0.75).double()\r\n ramda = sigma1_sq1 / (sigma1_sq1 + sigma2_sq1 + 1e-10)\r\n\r\n Q1 = (ramda * ssim_map2 + (1 - ramda) * ssim_map3) * bin_map\r\n Q2 = torch.max(ssim_map2, ssim_map3) * (1 - bin_map)\r\n Qy = (Q1 + Q2).mean().item()\r\n\r\n return Qy\r\n\r\ndef gaussian2d(n1, n2, sigma, device):\r\n x = torch.arange(-15, 16, device=device, dtype=torch.double)\r\n y = torch.arange(-15, 16, device=device, dtype=torch.double)\r\n x, y = torch.meshgrid(x, y)\r\n G = torch.exp(-(x**2 + y**2) / (2 * sigma**2)) / (2 * torch.pi * sigma**2)\r\n return G\r\n\r\ndef contrast(G1, G2, img):\r\n buff = F.conv2d(img.unsqueeze(0).unsqueeze(0), G1.unsqueeze(0).unsqueeze(0), padding=G1.shape[-1] // 2)\r\n buff1 = F.conv2d(img.unsqueeze(0).unsqueeze(0), G2.unsqueeze(0).unsqueeze(0), padding=G2.shape[-1] // 2)\r\n return buff / (buff1 + 1e-10) - 1\r\n\r\ndef Qcb_function(ir_img_tensor, vi_img_tensor, f_img_tensor):\r\n device = ir_img_tensor.device\r\n\r\n ir_img_tensor = ir_img_tensor.double().to(device)\r\n vi_img_tensor = vi_img_tensor.double().to(device)\r\n f_img_tensor = f_img_tensor.double().to(device)\r\n ir_img_tensor = (ir_img_tensor - ir_img_tensor.min()) / (ir_img_tensor.max() - ir_img_tensor.min())\r\n vi_img_tensor = (vi_img_tensor - vi_img_tensor.min()) / (vi_img_tensor.max() - vi_img_tensor.min())\r\n f_img_tensor = (f_img_tensor - f_img_tensor.min()) / (f_img_tensor.max() - f_img_tensor.min())\r\n\r\n f0 = 15.3870\r\n f1 = 1.3456\r\n a = 0.7622\r\n k = 1\r\n h = 1\r\n p = 3\r\n q = 2\r\n Z = 0.0001\r\n hang, lie = ir_img_tensor.shape[-2:]\r\n\r\n u, v = torch.meshgrid(torch.fft.fftfreq(hang, device=device), torch.fft.fftfreq(lie, device=device), indexing='ij')\r\n u = u * (hang / 30)\r\n v = v * (lie / 30)\r\n r = torch.sqrt(u**2 + v**2)\r\n Sd = (torch.exp(-(r / f0)**2) - a * torch.exp(-(r / f1)**2)).to(device)\r\n fim1 = torch.fft.ifft2(torch.fft.fft2(ir_img_tensor) * Sd).real\r\n fim2 = torch.fft.ifft2(torch.fft.fft2(vi_img_tensor) * Sd).real\r\n ffim = torch.fft.ifft2(torch.fft.fft2(f_img_tensor) * Sd).real\r\n G1 = gaussian2d(hang, lie, 2, device).to(device)\r\n G2 = gaussian2d(hang, lie, 4, device).to(device)\r\n C1 = contrast(G1, G2, fim1)\r\n C2 = contrast(G1, G2, fim2)\r\n Cf = contrast(G1, G2, ffim)\r\n C1P = (k * (torch.abs(C1)**p)) / (h * (torch.abs(C1)**q) + Z)\r\n C2P = (k * (torch.abs(C2)**p)) / (h * (torch.abs(C2)**q) + Z)\r\n CfP = (k * (torch.abs(Cf)**p)) / (h * (torch.abs(Cf)**q) + Z)\r\n\r\n mask1 = (C1P < CfP).double()\r\n Q1F = (C1P / CfP) * mask1 + (CfP / C1P) * (1 - mask1)\r\n mask2 = (C2P < CfP).double()\r\n Q2F = (C2P / CfP) * mask2 + (CfP / C2P) * (1 - mask2)\r\n ramda1 = (C1P**2) / (C1P**2 + C2P**2 + 1e-10)\r\n ramda2 = (C2P**2) / (C1P**2 + C2P**2 + 1e-10)\r\n Q = ramda1 * Q1F + ramda2 * Q2F\r\n Qcb = Q.mean().item()\r\n\r\n return Qcb\r\n\r\n" }, { "path": "Metric/Nabf.py", "content": "import numpy as np\r\nfrom scipy.signal import convolve2d\r\nimport math\r\nimport torch\r\ndef sobel_fn(x):\r\n vtemp = np.array([[-1, 0, 1], [-2, 0, 2], [-1, 0, 1]]) / 8\r\n htemp = np.array([[-1, -2, -1], [0, 0, 0], [1, 2, 1]]) / 8\r\n\r\n a, b = htemp.shape\r\n x_ext = per_extn_im_fn(x, a)\r\n p, q = x_ext.shape\r\n gv = np.zeros((p - 2, q - 2))\r\n gh = np.zeros((p - 2, q - 2))\r\n gv = convolve2d(x_ext, vtemp, mode='valid')\r\n gh = convolve2d(x_ext, htemp, mode='valid')\r\n\r\n return gv, gh\r\n\r\n\r\ndef per_extn_im_fn(x, wsize):\r\n\r\n hwsize = (wsize - 1) // 2 # Half window size excluding centre pixel.\r\n\r\n p, q = x.shape\r\n xout_ext = np.zeros((p + wsize - 1, q + wsize - 1))\r\n xout_ext[hwsize: p + hwsize, hwsize: q + hwsize] = x\r\n if wsize - 1 == hwsize + 1:\r\n xout_ext[0: hwsize, :] = xout_ext[2, :].reshape(1, -1)\r\n xout_ext[p + hwsize: p + wsize - 1, :] = xout_ext[-3, :].reshape(1, -1)\r\n\r\n xout_ext[:, 0: hwsize] = xout_ext[:, 2].reshape(-1, 1)\r\n xout_ext[:, q + hwsize: q + wsize - 1] = xout_ext[:, -3].reshape(-1, 1)\r\n\r\n return xout_ext\r\n\r\ndef get_Nabf(I1, I2, f):\r\n Td=2\r\n wt_min=0.001\r\n P=1\r\n Lg=1.5\r\n Nrg=0.9999\r\n kg=19\r\n sigmag=0.5\r\n Nra=0.9995\r\n ka=22\r\n sigmaa=0.5\r\n\r\n I1 = I1.cpu().numpy() if isinstance(I1, torch.Tensor) else I1\r\n I2 = I2.cpu().numpy() if isinstance(I2, torch.Tensor) else I2\r\n f = f.cpu().numpy() if isinstance(f, torch.Tensor) else f\r\n xrcw = f.astype(np.float64)\r\n x1 = I1.astype(np.float64)\r\n x2 = I2.astype(np.float64)\r\n\r\n gvA,ghA=sobel_fn(x1)\r\n gA=np.sqrt(ghA**2+gvA**2)\r\n\r\n gvB,ghB=sobel_fn(x2)\r\n gB=np.sqrt(ghB**2+gvB**2)\r\n\r\n gvF,ghF=sobel_fn(xrcw)\r\n gF=np.sqrt(ghF**2+gvF**2)\r\n\r\n gAF=np.zeros(gA.shape)\r\n gBF=np.zeros(gB.shape)\r\n aA=np.zeros(ghA.shape)\r\n aB=np.zeros(ghB.shape)\r\n aF=np.zeros(ghF.shape)\r\n p,q=xrcw.shape\r\n maskAF1 = (gA == 0) | (gF == 0)\r\n maskAF2 = (gA > gF)\r\n gAF[~maskAF1] = np.where(maskAF2, gF / gA, gA / gF)[~maskAF1]\r\n maskBF1 = (gB == 0) | (gF == 0)\r\n maskBF2 = (gB > gF)\r\n gBF[~maskBF1] = np.where(maskBF2, gF / gB, gB / gF)[~maskBF1]\r\n aA = np.where((gvA == 0) & (ghA == 0), 0, np.arctan(gvA / ghA))\r\n aB = np.where((gvB == 0) & (ghB == 0), 0, np.arctan(gvB / ghB))\r\n aF = np.where((gvF == 0) & (ghF == 0), 0, np.arctan(gvF / ghF))\r\n\r\n aAF=np.abs(np.abs(aA-aF)-np.pi/2)*2/np.pi\r\n aBF=np.abs(np.abs(aB-aF)-np.pi/2)*2/np.pi\r\n\r\n QgAF = Nrg / (1 + np.exp(-kg * (gAF - sigmag)))\r\n QaAF = Nra / (1 + np.exp(-ka * (aAF - sigmaa)))\r\n QAF = np.sqrt(QgAF * QaAF)\r\n QgBF = Nrg / (1 + np.exp(-kg * (gBF - sigmag)))\r\n QaBF = Nra / (1 + np.exp(-ka * (aBF - sigmaa)))\r\n QBF = np.sqrt(QgBF * QaBF)\r\n\r\n wtA = wt_min * np.ones((p, q))\r\n wtB = wt_min * np.ones((p, q))\r\n cA = np.ones((p, q))\r\n cB = np.ones((p, q))\r\n wtA = np.where(gA >= Td, cA * gA ** Lg, 0)\r\n wtB = np.where(gB >= Td, cB * gB ** Lg, 0)\r\n\r\n wt_sum = np.sum(wtA + wtB)\r\n QAF_wtsum = np.sum(QAF * wtA) / wt_sum\r\n QBF_wtsum = np.sum(QBF * wtB) / wt_sum\r\n QABF = QAF_wtsum + QBF_wtsum\r\n\r\n\r\n Qdelta = np.abs(QAF - QBF)\r\n QCinfo = (QAF + QBF - Qdelta) / 2\r\n QdeltaAF = QAF - QCinfo\r\n QdeltaBF = QBF - QCinfo\r\n QdeltaAF_wtsum = np.sum(QdeltaAF * wtA) / wt_sum\r\n QdeltaBF_wtsum = np.sum(QdeltaBF * wtB) / wt_sum\r\n QdeltaABF = QdeltaAF_wtsum + QdeltaBF_wtsum\r\n QCinfo_wtsum = np.sum(QCinfo * (wtA + wtB)) / wt_sum\r\n QABF11 = QdeltaABF + QCinfo_wtsum\r\n\r\n rr = np.zeros((p, q))\r\n rr = np.where(gF <= np.minimum(gA, gB), 1, 0)\r\n\r\n LABF = np.sum(rr * ((1 - QAF) * wtA + (1 - QBF) * wtB)) / wt_sum\r\n\r\n na1 = np.where((gF > gA) & (gF > gB), 2 - QAF - QBF, 0)\r\n NABF1 = np.sum(na1 * (wtA + wtB)) / wt_sum\r\n\r\n na = np.where((gF > gA) & (gF > gB), 1, 0)\r\n NABF = np.sum(na * ((1 - QAF) * wtA + (1 - QBF) * wtB)) / wt_sum\r\n return NABF" }, { "path": "Metric/Qabf.py", "content": "import numpy as np\r\nimport math\r\nfrom scipy.signal import convolve2d\r\n\r\n\r\ndef sobel_fn(x):\r\n vtemp = np.array([[-1, 0, 1], [-2, 0, 2], [-1, 0, 1]]) / 8\r\n htemp = np.array([[-1, -2, -1], [0, 0, 0], [1, 2, 1]]) / 8\r\n\r\n a, b = htemp.shape\r\n x_ext = per_extn_im_fn(x, a)\r\n p, q = x_ext.shape\r\n gv = np.zeros((p - 2, q - 2))\r\n gh = np.zeros((p - 2, q - 2))\r\n gv = convolve2d(x_ext, vtemp, mode='valid')\r\n gh = convolve2d(x_ext, htemp, mode='valid')\r\n\r\n return gv, gh\r\n\r\n\r\ndef per_extn_im_fn(x, wsize):\r\n hwsize = (wsize - 1) // 2\r\n\r\n p, q = x.shape\r\n xout_ext = np.zeros((p + wsize - 1, q + wsize - 1))\r\n xout_ext[hwsize: p + hwsize, hwsize: q + hwsize] = x\r\n\r\n\r\n if wsize - 1 == hwsize + 1:\r\n xout_ext[0: hwsize, :] = xout_ext[2, :].reshape(1, -1)\r\n xout_ext[p + hwsize: p + wsize - 1, :] = xout_ext[-3, :].reshape(1, -1)\r\n\r\n xout_ext[:, 0: hwsize] = xout_ext[:, 2].reshape(-1, 1)\r\n xout_ext[:, q + hwsize: q + wsize - 1] = xout_ext[:, -3].reshape(-1, 1)\r\n\r\n return xout_ext\r\n\r\ndef get_Qabf(pA, pB, pF):\r\n L = 1\r\n Tg = 0.9994\r\n kg = -15\r\n Dg = 0.5;\r\n Ta = 0.9879\r\n ka = -22\r\n Da = 0.8\r\n\r\n h1 = np.array([[1, 2, 1], [0, 0, 0], [-1, -2, -1]]).astype(np.float32)\r\n h2 = np.array([[0, 1, 2], [-1, 0, 1], [-2, -1, 0]]).astype(np.float32)\r\n h3 = np.array([[-1, 0, 1], [-2, 0, 2], [-1, 0, 1]]).astype(np.float32)\r\n\r\n\r\n strA = pA\r\n strB = pB\r\n strF = pF\r\n\r\n def flip180(arr):\r\n return np.flip(arr)\r\n\r\n def convolution(k, data):\r\n k = flip180(k)\r\n data = np.pad(data, ((1, 1), (1, 1)), 'constant', constant_values=(0, 0))\r\n img_new = convolve2d(data, k, mode='valid')\r\n return img_new\r\n\r\n def getArray(img):\r\n SAx = convolution(h3, img)\r\n SAy = convolution(h1, img)\r\n gA = np.sqrt(np.multiply(SAx, SAx) + np.multiply(SAy, SAy))\r\n n, m = img.shape\r\n aA = np.zeros((n, m))\r\n zero_mask = SAx == 0\r\n aA[~zero_mask] = np.arctan(SAy[~zero_mask] / SAx[~zero_mask])\r\n aA[zero_mask] = np.pi / 2\r\n return gA, aA\r\n\r\n gA, aA = getArray(strA)\r\n gB, aB = getArray(strB)\r\n gF, aF = getArray(strF)\r\n\r\n def getQabf(aA, gA, aF, gF):\r\n mask = (gA > gF)\r\n GAF = np.where(mask, gF / gA, np.where(gA == gF, gF, gA / gF))\r\n\r\n AAF = 1 - np.abs(aA - aF) / (math.pi / 2)\r\n\r\n QgAF = Tg / (1 + np.exp(kg * (GAF - Dg)))\r\n QaAF = Ta / (1 + np.exp(ka * (AAF - Da)))\r\n\r\n QAF = QgAF * QaAF\r\n return QAF\r\n\r\n QAF = getQabf(aA, gA, aF, gF)\r\n QBF = getQabf(aB, gB, aF, gF)\r\n\r\n deno = np.sum(gA + gB)\r\n nume = np.sum(np.multiply(QAF, gA) + np.multiply(QBF, gB))\r\n output = nume / deno\r\n return output" }, { "path": "Metric/eval_torch.py", "content": "import numpy as np\r\nfrom PIL import Image\r\nfrom Metric_torch import *\r\nfrom natsort import natsorted\r\nfrom tqdm import tqdm\r\nimport os\r\nimport torch\r\nimport warnings\r\nfrom openpyxl import Workbook, load_workbook\r\nfrom openpyxl.utils import get_column_letter\r\n\r\nwarnings.filterwarnings(\"ignore\")\r\ndevice = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\r\n\r\n\r\ndef write_excel(excel_name='metric.xlsx', worksheet_name='VIF', column_index=0, data=None):\r\n try:\r\n workbook = load_workbook(excel_name)\r\n except FileNotFoundError:\r\n workbook = Workbook()\r\n\r\n worksheet = workbook.create_sheet(title=worksheet_name) if worksheet_name not in workbook.sheetnames else workbook[\r\n worksheet_name]\r\n\r\n column = get_column_letter(column_index + 1)\r\n for i, value in enumerate(data):\r\n cell = worksheet[column + str(i + 1)]\r\n cell.value = value\r\n\r\n workbook.save(excel_name)\r\n\r\n\r\ndef evaluation_one(ir_name, vi_name, f_name):\r\n f_img = Image.open(f_name).convert('L')\r\n ir_img = Image.open(ir_name).convert('L')\r\n vi_img = Image.open(vi_name).convert('L')\r\n\r\n f_img_tensor = torch.tensor(np.array(f_img)).float().to(device)\r\n ir_img_tensor = torch.tensor(np.array(ir_img)).float().to(device)\r\n vi_img_tensor = torch.tensor(np.array(vi_img)).float().to(device)\r\n\r\n f_img_int = np.array(f_img).astype(np.int32)\r\n f_img_double = np.array(f_img).astype(np.float32)\r\n\r\n ir_img_int = np.array(ir_img).astype(np.int32)\r\n ir_img_double = np.array(ir_img).astype(np.float32)\r\n\r\n vi_img_int = np.array(vi_img).astype(np.int32)\r\n vi_img_double = np.array(vi_img).astype(np.float32)\r\n\r\n\r\n CE = CE_function(ir_img_tensor, vi_img_tensor, f_img_tensor)\r\n NMI = NMI_function(ir_img_int, vi_img_int, f_img_int, gray_level=256)\r\n QNCIE = QNCIE_function(ir_img_tensor, vi_img_tensor, f_img_tensor)\r\n TE = TE_function(ir_img_tensor, vi_img_tensor, f_img_tensor)\r\n EI = EI_function(f_img_tensor)\r\n Qy = Qy_function(ir_img_tensor, vi_img_tensor, f_img_tensor)\r\n Qcb = Qcb_function(ir_img_tensor, vi_img_tensor, f_img_tensor)\r\n EN = EN_function(f_img_tensor)\r\n MI = MI_function(ir_img_int, vi_img_int, f_img_int, gray_level=256)\r\n SF = SF_function(f_img_tensor)\r\n SD = SD_function(f_img_tensor)\r\n AG = AG_function(f_img_tensor)\r\n PSNR = PSNR_function(ir_img_tensor, vi_img_tensor, f_img_tensor)\r\n MSE = MSE_function(ir_img_tensor, vi_img_tensor, f_img_tensor)\r\n VIF = VIF_function(ir_img_tensor, vi_img_tensor, f_img_tensor)\r\n CC = CC_function(ir_img_tensor, vi_img_tensor, f_img_tensor)\r\n SCD = SCD_function(ir_img_tensor, vi_img_tensor, f_img_tensor)\r\n Qabf = Qabf_function(ir_img_double, vi_img_double, f_img_double)\r\n Nabf = Nabf_function(ir_img_tensor, vi_img_tensor, f_img_tensor)\r\n SSIM = SSIM_function(ir_img_double, vi_img_double, f_img_double)\r\n MS_SSIM = MS_SSIM_function(ir_img_double, vi_img_double, f_img_double)\r\n\r\n return CE, NMI, QNCIE, TE, EI, Qy, Qcb, EN, MI, SF, AG, SD, CC, SCD, VIF, MSE, PSNR, Qabf, Nabf, SSIM, MS_SSIM\r\n\r\n\r\nif __name__ == '__main__':\r\n if __name__ == '__main__':\r\n with_mean = True\r\n config = {\r\n 'dataroot': '/mnt/disk1/IVIF/', # Change to your local infrared and visible images path\r\n 'results_root': '/mnt/disk1/IVIF/', # Change to your local fusion images path\r\n 'dataset': 'M3FD_4200', # Specify the dataset name\r\n 'save_dir': '/mnt/disk4/test' # Directory for saving metrics\r\n }\r\n\r\n ir_dir = os.path.join(config['dataroot'], config['dataset'], 'Ir') # Infrared images directory\r\n vi_dir = os.path.join(config['dataroot'], config['dataset'], 'Vis') # Visible images directory\r\n f_dir = os.path.join(config['results_root'], config['dataset']) # Fusion images directory\r\n os.makedirs(config['save_dir'], exist_ok=True)\r\n filelist = natsorted(os.listdir(ir_dir))[:300]\r\n metric_save_name = os.path.join(config['save_dir'], f'metric_{config[\"dataset\"]}.xlsx') # Metrics file name\r\n\r\n # Change to the directory name of the fusion images you want to evaluate\r\n Method_list = [\r\n 'BDLFusion', 'CAF', 'CDDFuse', 'CoCoNet', 'DATFuse', 'DDcGAN', 'DDFM',\r\n 'DeFusion', 'Densefuse', 'DIDFuse', 'EMMA', 'FusinDN', 'GANMcC',\r\n 'IF-FILM', 'IGNet', 'IRFS', 'LRRNet', 'MetaFusion', 'MFEIF', 'MRFS',\r\n 'PAIF', 'PMGI', 'PSFusion', 'ReCoNet', 'RFN-Nest', 'SDCFusion',\r\n 'SDNet', 'SeAFusion', 'SegMif', 'SHIP', 'SuperFusion', 'SwinFusion',\r\n 'TarDAL', 'Text-IF', 'TGFuse', 'TIMFusion', 'U2Fusion', 'UMFusion',\r\n 'YDTR', 'FusionGAN', 'DetFusion', 'MoE-Fusion', 'PromptF'\r\n ]\r\n\r\n # Starting index for the method 'BDLFusion'\r\n start_index = Method_list.index('BDLFusion')\r\n\r\n for i, Method in enumerate(Method_list[start_index:], start=start_index):\r\n CE_list = []\r\n NMI_list = []\r\n QNCIE_list = []\r\n TE_list = []\r\n EI_list = []\r\n Qy_list = []\r\n Qcb_list = []\r\n EN_list = []\r\n MI_list = []\r\n SF_list = []\r\n AG_list = []\r\n SD_list = []\r\n CC_list = []\r\n SCD_list = []\r\n VIF_list = []\r\n MSE_list = []\r\n PSNR_list = []\r\n Qabf_list = []\r\n Nabf_list = []\r\n SSIM_list = []\r\n MS_SSIM_list = []\r\n filename_list = ['']\r\n sub_f_dir = os.path.join(f_dir, Method)\r\n eval_bar = tqdm(filelist)\r\n for _, item in enumerate(eval_bar):\r\n ir_name = os.path.join(ir_dir, item)\r\n vi_name = os.path.join(vi_dir, item)\r\n f_name = os.path.join(sub_f_dir, item)\r\n\r\n if os.path.exists(f_name):\r\n print(ir_name, vi_name, f_name)\r\n CE, NMI, QNCIE, TE, EI, Qy, Qcb, EN, MI, SF, AG, SD, CC, SCD, VIF, MSE, PSNR, Qabf, Nabf, SSIM, MS_SSIM = evaluation_one(ir_name, vi_name, f_name)\r\n CE_list.append(CE)\r\n NMI_list.append(NMI)\r\n QNCIE_list.append(QNCIE)\r\n TE_list.append(TE)\r\n EI_list.append(EI)\r\n Qy_list.append(Qy)\r\n Qcb_list.append(Qcb)\r\n EN_list.append(EN)\r\n MI_list.append(MI)\r\n SF_list.append(SF)\r\n AG_list.append(AG)\r\n SD_list.append(SD)\r\n CC_list.append(CC)\r\n SCD_list.append(SCD)\r\n VIF_list.append(VIF)\r\n MSE_list.append(MSE)\r\n PSNR_list.append(PSNR)\r\n Qabf_list.append(Qabf)\r\n Nabf_list.append(Nabf)\r\n SSIM_list.append(SSIM)\r\n MS_SSIM_list.append(MS_SSIM)\r\n filename_list.append(item)\r\n eval_bar.set_description(\"{} | {}\".format(Method, item))\r\n\r\n if with_mean:\r\n CE_tensor = torch.tensor(CE_list).mean().item()\r\n CE_list.append(CE_tensor)\r\n NMI_tensor = torch.tensor(NMI_list).mean().item()\r\n NMI_list.append(NMI_tensor)\r\n QNCIE_tensor = torch.tensor(QNCIE_list).mean().item()\r\n QNCIE_list.append(QNCIE_tensor)\r\n TE_tensor = torch.tensor(TE_list).mean().item()\r\n TE_list.append(TE_tensor)\r\n EI_tensor = torch.tensor(EI_list).mean().item()\r\n EI_list.append(EI_tensor)\r\n Qy_tensor = torch.tensor(Qy_list).mean().item()\r\n Qy_list.append(Qy_tensor)\r\n Qcb_tensor = torch.tensor(Qcb_list).mean().item()\r\n Qcb_list.append(Qcb_tensor)\r\n EN_tensor = torch.tensor(EN_list).mean().item()\r\n EN_list.append(EN_tensor)\r\n MI_tensor = torch.tensor(MI_list).mean().item()\r\n MI_list.append(MI_tensor)\r\n SF_tensor = torch.tensor(SF_list).mean().item()\r\n SF_list.append(SF_tensor)\r\n AG_tensor = torch.tensor(AG_list).mean().item()\r\n AG_list.append(AG_tensor)\r\n SD_tensor = torch.tensor(SD_list).mean().item()\r\n SD_list.append(SD_tensor)\r\n CC_tensor = torch.tensor(CC_list).mean().item()\r\n CC_list.append(CC_tensor)\r\n SCD_tensor = torch.tensor(SCD_list).mean().item()\r\n SCD_list.append(SCD_tensor)\r\n VIF_tensor = torch.tensor(VIF_list).mean().item()\r\n VIF_list.append(VIF_tensor)\r\n MSE_tensor = torch.tensor(MSE_list).mean().item()\r\n MSE_list.append(MSE_tensor)\r\n PSNR_tensor = torch.tensor(PSNR_list).mean().item()\r\n PSNR_list.append(PSNR_tensor)\r\n Qabf_list.append(np.mean(Qabf_list))\r\n Nabf_tensor = torch.tensor(Nabf_list).mean().item()\r\n Nabf_list.append(Nabf_tensor)\r\n SSIM_tensor = torch.tensor(SSIM_list).mean().item()\r\n SSIM_list.append(SSIM_tensor)\r\n MS_SSIM_tensor = torch.tensor(MS_SSIM_list).mean().item()\r\n MS_SSIM_list.append(MS_SSIM_tensor)\r\n filename_list.append('mean')\r\n\r\n\r\n CE_list.insert(0, '{}'.format(Method))\r\n NMI_list.insert(0, '{}'.format(Method))\r\n QNCIE_list.insert(0, '{}'.format(Method))\r\n TE_list.insert(0, '{}'.format(Method))\r\n EI_list.insert(0, '{}'.format(Method))\r\n Qy_list.insert(0, '{}'.format(Method))\r\n Qcb_list.insert(0, '{}'.format(Method))\r\n EN_list.insert(0, '{}'.format(Method))\r\n MI_list.insert(0, '{}'.format(Method))\r\n SF_list.insert(0, '{}'.format(Method))\r\n AG_list.insert(0, '{}'.format(Method))\r\n SD_list.insert(0, '{}'.format(Method))\r\n CC_list.insert(0, '{}'.format(Method))\r\n SCD_list.insert(0, '{}'.format(Method))\r\n VIF_list.insert(0, '{}'.format(Method))\r\n MSE_list.insert(0, '{}'.format(Method))\r\n PSNR_list.insert(0, '{}'.format(Method))\r\n Qabf_list.insert(0, '{}'.format(Method))\r\n Nabf_list.insert(0, '{}'.format(Method))\r\n SSIM_list.insert(0, '{}'.format(Method))\r\n MS_SSIM_list.insert(0, '{}'.format(Method))\r\n\r\n if i == start_index:\r\n write_excel(metric_save_name, 'CE', 0, filename_list)\r\n write_excel(metric_save_name, 'NMI', 0, filename_list)\r\n write_excel(metric_save_name, 'QNCIE', 0, filename_list)\r\n write_excel(metric_save_name, 'TE', 0, filename_list)\r\n write_excel(metric_save_name, 'EI', 0, filename_list)\r\n write_excel(metric_save_name, 'Qy', 0, filename_list)\r\n write_excel(metric_save_name, 'Qcb', 0, filename_list)\r\n write_excel(metric_save_name, 'EN', 0, filename_list)\r\n write_excel(metric_save_name, \"MI\", 0, filename_list)\r\n write_excel(metric_save_name, \"SF\", 0, filename_list)\r\n write_excel(metric_save_name, \"AG\", 0, filename_list)\r\n write_excel(metric_save_name, \"SD\", 0, filename_list)\r\n write_excel(metric_save_name, \"CC\", 0, filename_list)\r\n write_excel(metric_save_name, \"SCD\", 0, filename_list)\r\n write_excel(metric_save_name, \"VIF\", 0, filename_list)\r\n write_excel(metric_save_name, \"MSE\", 0, filename_list)\r\n write_excel(metric_save_name, \"PSNR\", 0, filename_list)\r\n write_excel(metric_save_name, \"Qabf\", 0, filename_list)\r\n write_excel(metric_save_name, \"Nabf\", 0, filename_list)\r\n write_excel(metric_save_name, \"SSIM\", 0, filename_list)\r\n write_excel(metric_save_name, \"MS_SSIM\", 0, filename_list)\r\n\r\n write_excel(metric_save_name, 'CE', i + 1,\r\n [x.item() if isinstance(x, torch.Tensor) else float(x) if isinstance(x, (int, float)) else x for x\r\n in CE_list])\r\n write_excel(metric_save_name, 'NMI', i + 1,\r\n [x.item() if isinstance(x, torch.Tensor) else float(x) if isinstance(x, (int, float)) else x for x\r\n in NMI_list])\r\n write_excel(metric_save_name, 'QNCIE', i + 1,\r\n [x.item() if isinstance(x, torch.Tensor) else float(x) if isinstance(x, (int, float)) else x for x\r\n in QNCIE_list])\r\n write_excel(metric_save_name, 'TE', i + 1,\r\n [x.item() if isinstance(x, torch.Tensor) else float(x) if isinstance(x, (int, float)) else x for x\r\n in TE_list])\r\n write_excel(metric_save_name, 'EI', i + 1,\r\n [x.item() if isinstance(x, torch.Tensor) else float(x) if isinstance(x, (int, float)) else x for x\r\n in EI_list])\r\n write_excel(metric_save_name, 'Qy', i + 1,\r\n [x.item() if isinstance(x, torch.Tensor) else float(x) if isinstance(x, (int, float)) else x for x\r\n in Qy_list])\r\n write_excel(metric_save_name, 'Qcb', i + 1,\r\n [x.item() if isinstance(x, torch.Tensor) else float(x) if isinstance(x, (int, float)) else x for x\r\n in Qcb_list])\r\n write_excel(metric_save_name, 'EN', i + 1,\r\n [x.item() if isinstance(x, torch.Tensor) else float(x) if isinstance(x, (int, float)) else x for x\r\n in EN_list])\r\n write_excel(metric_save_name, 'MI', i + 1,\r\n [x.item() if isinstance(x, torch.Tensor) else float(x) if isinstance(x, (int, float)) else x for x\r\n in MI_list])\r\n write_excel(metric_save_name, 'SF', i + 1,\r\n [x.item() if isinstance(x, torch.Tensor) else float(x) if isinstance(x, (int, float)) else x for x\r\n in SF_list])\r\n write_excel(metric_save_name, 'AG', i + 1,\r\n [x.item() if isinstance(x, torch.Tensor) else float(x) if isinstance(x, (int, float)) else x for x\r\n in AG_list])\r\n write_excel(metric_save_name, 'SD', i + 1,\r\n [x.item() if isinstance(x, torch.Tensor) else float(x) if isinstance(x, (int, float)) else x for x\r\n in SD_list])\r\n write_excel(metric_save_name, 'CC', i + 1,\r\n [x.item() if isinstance(x, torch.Tensor) else float(x) if isinstance(x, (int, float)) else x for x\r\n in CC_list])\r\n write_excel(metric_save_name, 'SCD', i + 1,\r\n [x.item() if isinstance(x, torch.Tensor) else float(x) if isinstance(x, (int, float)) else x for x\r\n in SCD_list])\r\n write_excel(metric_save_name, 'VIF', i + 1,\r\n [x.item() if isinstance(x, torch.Tensor) else float(x) if isinstance(x, (int, float)) else x for x\r\n in VIF_list])\r\n write_excel(metric_save_name, 'MSE', i + 1,\r\n [x.item() if isinstance(x, torch.Tensor) else float(x) if isinstance(x, (int, float)) else x for x\r\n in MSE_list])\r\n write_excel(metric_save_name, 'PSNR', i + 1,\r\n [x.item() if isinstance(x, torch.Tensor) else float(x) if isinstance(x, (int, float)) else x for x\r\n in PSNR_list])\r\n write_excel(metric_save_name, 'Qabf', i + 1, Qabf_list)\r\n write_excel(metric_save_name, 'Nabf', i + 1,\r\n [x.item() if isinstance(x, torch.Tensor) else float(x) if isinstance(x, (int, float)) else x for x\r\n in Nabf_list])\r\n write_excel(metric_save_name, 'SSIM', i + 1,\r\n [x.item() if isinstance(x, torch.Tensor) else float(x) if isinstance(x, (int, float)) else x for x\r\n in SSIM_list])\r\n write_excel(metric_save_name, 'MS_SSIM', i + 1,\r\n [x.item() if isinstance(x, torch.Tensor) else float(x) if isinstance(x, (int, float)) else x for x\r\n in MS_SSIM_list])\r\n" }, { "path": "Metric/ssim.py", "content": "import warnings\r\nimport torch\r\nimport torch.nn as nn\r\nimport torch.nn.functional as F\r\nimport torchvision.transforms.functional as TF\r\nimport numpy as np\r\n\r\ndef _fspecial_gauss_1d(size, sigma):\r\n coords = torch.arange(size, dtype=torch.float32)\r\n coords -= size // 2\r\n\r\n g = torch.exp(-(coords ** 2) / (2 * sigma ** 2))\r\n g /= g.sum()\r\n\r\n return g.unsqueeze(0).unsqueeze(0)\r\n\r\n\r\ndef gaussian_filter(input, win):\r\n assert all([ws == 1 for ws in win.shape[1:-1]]), win.shape\r\n if len(input.shape) == 4:\r\n conv = F.conv2d\r\n elif len(input.shape) == 5:\r\n conv = F.conv3d\r\n else:\r\n raise NotImplementedError(input.shape)\r\n\r\n C = input.shape[1]\r\n out = input\r\n for i, s in enumerate(input.shape[2:]):\r\n if s >= win.shape[-1]:\r\n perms = list(range(win.ndim))\r\n perms[2 + i] = perms[-1]\r\n perms[-1] = 2 + i\r\n out = conv(out, weight=win.permute(perms), stride=1, padding=0, groups=C)\r\n else:\r\n warnings.warn(\r\n f\"Skipping Gaussian Smoothing at dimension 2+{i} for input: {input.shape} and win size: {win.shape[-1]}\"\r\n )\r\n\r\n return out\r\n\r\n\r\ndef _ssim(X, Y, data_range, win, K=(0.01, 0.03)):\r\n K1, K2 = K\r\n compensation = 1.0\r\n\r\n C1 = (K1 * data_range) ** 2\r\n C2 = (K2 * data_range) ** 2\r\n\r\n win = win.type_as(X)\r\n\r\n mu1 = gaussian_filter(X, win)\r\n mu2 = gaussian_filter(Y, win)\r\n\r\n mu1_sq = mu1.pow(2)\r\n mu2_sq = mu2.pow(2)\r\n mu1_mu2 = mu1 * mu2\r\n\r\n sigma1_sq = compensation * (gaussian_filter(X * X, win) - mu1_sq)\r\n sigma2_sq = compensation * (gaussian_filter(Y * Y, win) - mu2_sq)\r\n sigma12 = compensation * (gaussian_filter(X * Y, win) - mu1_mu2)\r\n\r\n cs_map = (2 * sigma12 + C2) / (sigma1_sq + sigma2_sq + C2)\r\n ssim_map = ((2 * mu1_mu2 + C1) / (mu1_sq + mu2_sq + C1)) * cs_map\r\n\r\n ssim_per_channel = torch.flatten(ssim_map, 2).mean(-1)\r\n cs = torch.flatten(cs_map, 2).mean(-1)\r\n return ssim_per_channel, cs\r\n\r\ndef ssim(X,\r\n Y,\r\n data_range=255,\r\n size_average=True,\r\n win_size=11,\r\n win_sigma=1.5,\r\n win=None,\r\n K=(0.01, 0.03),\r\n nonnegative_ssim=False):\r\n X = TF.to_tensor(X).unsqueeze(0).unsqueeze(0)\r\n Y = TF.to_tensor(Y).unsqueeze(0).unsqueeze(0)\r\n if not X.shape == Y.shape:\r\n raise ValueError(\"Input images should have the same dimensions.\")\r\n\r\n for d in range(len(X.shape) - 1, 1, -1):\r\n X = torch.squeeze(X, dim=d)\r\n Y = torch.squeeze(Y, dim=d)\r\n\r\n if len(X.shape) not in (4, 5):\r\n raise ValueError(f\"Input images should be 4-d or 5-d tensors, but got {X.shape}\")\r\n\r\n if not X.dtype == Y.dtype:\r\n raise ValueError(\"Input images should have the same dtype.\")\r\n\r\n if win is not None: # set win_size\r\n win_size = win.shape[-1]\r\n\r\n if not (win_size % 2 == 1):\r\n raise ValueError(\"Window size should be odd.\")\r\n\r\n if win is None:\r\n win = _fspecial_gauss_1d(win_size, win_sigma)\r\n win = win.repeat([X.shape[1]] + [1] * (len(X.shape) - 1))\r\n\r\n ssim_per_channel, _ = _ssim(X, Y, data_range=data_range, win=win, K=K)\r\n if nonnegative_ssim:\r\n ssim_per_channel = F.relu(ssim_per_channel)\r\n\r\n if size_average:\r\n return ssim_per_channel.mean()\r\n else:\r\n return ssim_per_channel.mean(dim=1)\r\n\r\n\r\ndef ms_ssim(\r\n X,\r\n Y,\r\n data_range=255,\r\n size_average=True,\r\n win_size=11,\r\n win_sigma=1.5,\r\n win=None,\r\n weights=None,\r\n K=(0.01, 0.03)\r\n ):\r\n X = TF.to_tensor(X).unsqueeze(0).unsqueeze(0)\r\n Y = TF.to_tensor(Y).unsqueeze(0).unsqueeze(0)\r\n if not X.shape == Y.shape:\r\n raise ValueError(\"Input images should have the same dimensions.\")\r\n\r\n for d in range(len(X.shape) - 1, 1, -1):\r\n X = X.squeeze(dim=d)\r\n Y = Y.squeeze(dim=d)\r\n\r\n if not X.dtype == Y.dtype:\r\n raise ValueError(\"Input images should have the same dtype.\")\r\n\r\n if len(X.shape) == 4:\r\n avg_pool = F.avg_pool2d\r\n elif len(X.shape) == 5:\r\n avg_pool = F.avg_pool3d\r\n else:\r\n raise ValueError(f\"Input images should be 4-d or 5-d tensors, but got {X.shape}\")\r\n\r\n if win is not None:\r\n win_size = win.shape[-1]\r\n\r\n if not (win_size % 2 == 1):\r\n raise ValueError(\"Window size should be odd.\")\r\n\r\n smaller_side = min(X.shape[-2:])\r\n assert smaller_side > (win_size - 1) * (\r\n 2 ** 4\r\n ), \"Image size should be larger than %d due to the 4 downsamplings in ms-ssim\" % ((win_size - 1) * (2 ** 4))\r\n\r\n if weights is None:\r\n weights = [0.0448, 0.2856, 0.3001, 0.2363, 0.1333]\r\n weights = torch.tensor(weights, dtype=X.dtype)\r\n\r\n if win is None:\r\n win = _fspecial_gauss_1d(win_size, win_sigma)\r\n win = win.repeat([X.shape[1]] + [1] * (len(X.shape) - 1))\r\n\r\n levels = weights.shape[0]\r\n mcs = []\r\n for i in range(levels):\r\n ssim_per_channel, cs = _ssim(X, Y, win=win, data_range=data_range, K=K)\r\n\r\n if i < levels - 1:\r\n mcs.append(F.relu(cs))\r\n padding = [s % 2 for s in X.shape[2:]]\r\n X = avg_pool(X, kernel_size=2, padding=padding)\r\n Y = avg_pool(Y, kernel_size=2, padding=padding)\r\n\r\n ssim_per_channel = F.relu(ssim_per_channel)\r\n mcs_and_ssim = torch.stack(mcs + [ssim_per_channel], dim=0)\r\n ms_ssim_val = torch.prod(mcs_and_ssim ** weights.reshape((-1, 1, 1)), dim=0)\r\n\r\n if size_average:\r\n return ms_ssim_val.mean()\r\n else:\r\n return ms_ssim_val.mean(dim=1)\r\n\r\nclass SSIM(nn.Module):\r\n def __init__(\r\n self,\r\n data_range=255,\r\n size_average=True,\r\n win_size=11,\r\n win_sigma=1.5,\r\n channel=3,\r\n spatial_dims=2,\r\n K=(0.01, 0.03),\r\n nonnegative_ssim=False,\r\n ):\r\n super(SSIM, self).__init__()\r\n self.win_size = win_size\r\n self.win = _fspecial_gauss_1d(win_size, win_sigma).tile([channel, 1] + [1] * spatial_dims)\r\n self.size_average = size_average\r\n self.data_range = data_range\r\n self.K = K\r\n self.nonnegative_ssim = nonnegative_ssim\r\n\r\n def forward(self, X, Y):\r\n return ssim(\r\n X,\r\n Y,\r\n data_range=self.data_range,\r\n size_average=self.size_average,\r\n win=self.win,\r\n K=self.K,\r\n nonnegative_ssim=self.nonnegative_ssim,\r\n ).item()\r\n\r\n\r\nclass MS_SSIM(nn.Module):\r\n def __init__(\r\n self,\r\n data_range=255,\r\n size_average=True,\r\n win_size=11,\r\n win_sigma=1.5,\r\n channel=3,\r\n spatial_dims=2,\r\n weights=None,\r\n K=(0.01, 0.03),\r\n ):\r\n super(MS_SSIM, self).__init__()\r\n self.win_size = win_size\r\n self.win = _fspecial_gauss_1d(win_size, win_sigma).tile([channel, 1] + [1] * spatial_dims)\r\n self.size_average = size_average\r\n self.data_range = data_range\r\n self.weights = weights\r\n self.K = K\r\n\r\n def forward(self, X, Y):\r\n return ms_ssim(\r\n X,\r\n Y,\r\n data_range=self.data_range,\r\n size_average=self.size_average,\r\n win=self.win,\r\n weights=self.weights,\r\n K=self.K,\r\n\r\n ).item()\r\n" }, { "path": "README.md", "content": "\n## Latest News 🔥🔥\n[2024-12-12] Our survey paper [__Infrared and Visible Image Fusion: From Data Compatibility to Task Adaption.__] has been accepted by IEEE Transactions on Pattern Analysis and Machine Intelligence!\n([Paper](https://ieeexplore.ieee.org/abstract/document/10812907))([中文版](https://pan.baidu.com/s/1EIRYSULa-pd2FRmIdG693g?pwd=aiey))\n\n[2026-04-15] We have updated the repository with state-of-the-art methods for both Image Fusion and Video Fusion.\n\n# IVIF Zoo\nWelcome to IVIF Zoo, a comprehensive repository dedicated to Infrared and Visible Image Fusion (IVIF). Based on our survey paper [__Infrared and Visible Image Fusion: From Data Compatibility to Task Adaption.__ *Jinyuan Liu, Guanyao Wu, Zhu Liu, Di Wang, Zhiying Jiang, Long Ma, Wei Zhong, Xin Fan, Risheng Liu**], this repository aims to serve as a central hub for researchers, engineers, and enthusiasts in the field of IVIF. Here, you'll find a wide array of resources, tools, and datasets, curated to accelerate advancements and foster collaboration in infrared-visible image fusion technologies.\n\n***\n\n![preview](assets/light2.png)\nA detailed spectrogram depicting almost all wavelength and frequency ranges, particularly expanding the range of the human visual system and annotating corresponding computer vision and image fusion datasets.\n\n![preview](assets/pipeline1.png)\nThe diagram of infrared and visible image fusion for practical applications. Existing image fusion methods majorly focus on the design of architectures and training strategies for visual enhancement, few considering the adaptation for downstream visual perception tasks. Additionally, from the data compatibility perspective, pixel misalignment and adversarial attacks of image fusion are two major challenges. Additionally, integrating comprehensive semantic information for tasks like semantic segmentation, object detection, and salient object detection remains underexplored, posing a critical obstacle in image fusion.\n\n![preview](assets/sankey1.png)\n A classification sankey diagram containing typical fusion methods.\n\n***\n\n## 导航(Navigation)\n\n- [数据集 (Datasets)](#数据集datasets)\n- [方法集 (Method Set)](#方法集method-set)\n - [纯融合方法 (Fusion for Visual Enhancement)](#纯融合方法fusion-for-visual-enhancement)\n - [数据兼容方法 (Data Compatible)](#数据兼容方法data-compatible)\n - [面向应用方法 (Application-oriented)](#面向应用方法application-oriented)\n- [评价指标 (Evaluation Metric)](#评价指标evaluation-metric)\n### [🔥🚀资源库 (Resource Library)](#资源库resource-library) \n`It covers all results of our survey paper, available for download from Baidu Cloud.`\n - 💥[融合 (Fusion)](#融合fusion) \n - ✂️[分割 (Segmentation)](#分割segmentation) `Based on SegFormer`\n - 🔍[检测 (Detection)](#检测detection) `Based on YOLO-v5`\n - [计算效率 (Computational Efficiency)](#计算效率computational-efficiency)\n# 数据集(Datasets)\n## 图像数据集（Image Datasets）\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
DatasetImg pairsResolutionColorObj/CatsCha-ScAnnoDownLoad
TNO261768×576❌few✔❌Link
RoadScene 🔥221Various✔medium❌❌Link
VIFB21VariousVariousfew❌❌Link
MS2999768×576✔14146 / 6❌✔Link
LLVIP168361280×720✔pedestrian / 1❌✔Link
M3FD 🔥42001024×768✔33603 / 6✔✔Link
MFNet1569640×480✔abundant / 8❌✔Link
FMB 🔥1500800×600✔abundant / 14❌✔Link
\n\n## 视频数据集（Video Datasets）\n\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
DatasetVideo CountTotal FramesResolutionDownLoad
VF-Bench797Over 200,0002K/540p/480pLink
HDO247,500640×480Link
M3SVD220153,797640×480Link
\n\n\nIf the M3FD and FMB datasets are helpful to you, please cite the following paper:\n\n```\n@inproceedings{liu2022target,\n title={Target-aware dual adversarial learning and a multi-scenario multi-modality benchmark to fuse infrared and visible for object detection},\n author={Liu, Jinyuan and Fan, Xin and Huang, Zhanbo and Wu, Guanyao and Liu, Risheng and Zhong, Wei and Luo, Zhongxuan},\n booktitle={Proceedings of the IEEE/CVF conference on computer vision and pattern recognition},\n pages={5802--5811},\n year={2022}\n}\n```\n\n```\n@inproceedings{liu2023multi,\n title={Multi-interactive feature learning and a full-time multi-modality benchmark for image fusion and segmentation},\n author={Liu, Jinyuan and Liu, Zhu and Wu, Guanyao and Ma, Long and Liu, Risheng and Zhong, Wei and Luo, Zhongxuan and Fan, Xin},\n booktitle={Proceedings of the IEEE/CVF international conference on computer vision},\n pages={8115--8124},\n year={2023}\n}\n```\n\n# 方法集(Method Set)\n## 纯融合方法(Fusion for Visual Enhancement)\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
Aspects
(分类)		Methods
(方法)		Title
(标题)		Venue
(发表场所)		Source
(资源)
Auto-EncoderDenseFuseDensefuse: A fusion approach to infrared and visible imagesTIP '18Paper/Code
Auto-EncoderSEDRFuseSedrfuse: A symmetric encoder–decoder with residual block network for infrared and visible image fusionTIM '20Paper/Code
Auto-EncoderDIDFuseDidfuse: Deep image decomposition for infrared and visible image fusionIJCAI '20Paper/Code
Auto-EncoderMFEIFLearning a deep multi-scale feature ensemble and an edge-attention guidance for image fusionTCSVT '21Paper/Code
Auto-EncoderRFN-NestRfn-nest: An end-to-end residual fusion network for infrared and visible imagesTIM '21Paper/Code
Auto-EncoderSFAFuseSelf-supervised feature adaption for infrared and visible image fusionInfFus '21Paper/Code
Auto-EncoderSMoASmoa: Searching a modality-oriented architecture for infrared and visible image fusionSPL '21Paper/Code
Auto-EncoderRe2FusionRes2fusion: Infrared and visible image fusion based on dense res2net and double nonlocal attention modelsTIM '22Paper/Code
Auto-EncoderRPFNetResidual Prior-driven Frequency-aware Network for Image FusionACM MM '25Paper/Code
Auto-EncoderTTDTest-Time Dynamic Image FusionNeurIPS '24Paper/Code
GANFusionGANFusiongan: A generative adversarial network for infrared and visible image fusionInfFus '19Paper/Code
GANDDcGANLearning a generative model for fusing infrared and visible images via conditional generative adversarial network with dual discriminatorsTIP '19Paper/Code
GANAtFGANAttentionfgan: Infrared and visible image fusion using attention-based generative adversarial networksTMM '20Paper
GANDPALInfrared and visible image fusion via detail preserving adversarial learningInfFus '20Paper/Code
GAND2WGANInfrared and visible image fusion using dual discriminators generative adversarial networks with wasserstein distanceInfSci '20Paper
GANGANMcCGanmcc: A generative adversarial network with multiclassification constraints for infrared and visible image fusionTIM '20Paper/Code
GANICAFusionInfrared and visible image fusion via interactive compensatory attention adversarial learningTMM '22Paper/Code
GANTCGANTransformer based conditional gan for multimodal image fusionTMM '23Paper/Code
GANDCFusionDCFusion: A Dual-Frequency Cross-Enhanced Fusion Network for Infrared and Visible Image FusionTIM '23Paper
GANFreqGANFreqgan: Infrared and visible image fusion via unified frequency adversarial learningTCSVT '24Paper/Code
GANDDBFDispel Darkness for Better Fusion: A Controllable Visual Enhancer based on Cross-modal Conditional Adversarial LearningCVPR '24Paper/Code
GANCCFConditional Controllable Image FusionNeurIPS '24Paper/Code
CNNBIMDLA bilevel integrated model with data-driven layer ensemble for multi-modality image fusionTIP '20Paper
CNNMgAN-FuseMultigrained attention network for infrared and visible image fusionTIM '20Paper
CNNAUIFEfficient and model-based infrared and visible image fusion via algorithm unrollingTCSVT '21Paper/Code
CNNRXDNFuseRxdnfuse: A aggregated residual dense network for infrared and visible image fusionInfFus '21Paper
CNNSTDFusionNetStdfusionnet: An infrared and visible image fusion network based on salient target detectionTIM '21Paper/Code
CNNCUFDCufd: An encoder–decoder network for visible and infrared image fusion based on common and unique feature decompositionCVIU '22Paper/Code
CNNDif-FusionDif-fusion: Towards high color fidelity in infrared and visible image fusion with diffusion modelsTIP '23Paper/Code
CNNL2NetL2Net: Infrared and Visible Image Fusion Using Lightweight Large Kernel Convolution NetworkTIP '23Paper/Code
CNNIGNetLearning a graph neural network with cross modality interaction for image fusionACMMM '23Paper/Code
CNNLRRNetLrrnet: A novel representation learning guided fusion network for infrared and visible imagesTPAMI '23Paper/Code
CNNMetaFusionMetafusion: Infrared and visible image fusion via meta-feature embedding from object detectionCVPR '23Paper/Code
CNNPSFusionRethinking the necessity of image fusion in high-level vision tasks: A practical infrared and visible image fusion network based on progressive semantic injection and scene fidelityInfFus '23Paper/Code
CNNLUT-FuseLUT-Fuse: Towards Extremely Fast Infrared and Visible Image Fusion via Distillation to Learnable Look-Up TablesICCV '25Paper/Code
CNNPMAINetProgressive Modality-Adaptive Interactive Network for Multi-Modality Image FusionIJCAI '25Paper
TransformerSwinFusionSwinfusion: Cross-domain long-range learning for general image fusion via swin transformerJAS '22Paper/Code
TransformerYDTRYdtr: Infrared and visible image fusion via y-shape dynamic transformerTMM '22Paper/Code
TransformerIFTImage fusion transformerICIP '22Paper/Code
TransformerCDDFuseCddfuse: Correlation-driven dual-branch feature decomposition for multi-modality image fusionCVPR '23Paper/Code
TransformerTGFuseTgfuse: An infrared and visible image fusion approach based on transformer and generative adversarial networkTIP '23Paper/Code
TransformerCMTFusionCross-modal transformers for infrared and visible image fusionTCSVT '23Paper/Code
TransformerText-IFText-if: Leveraging semantic text guidance for degradation-aware and interactive image fusionCVPR '24Paper/Code
TransformerPromptFPromptfusion: Harmonized semantic prompt learning for infrared and visible image fusionJAS '24
TransformerMaeFuseMaeFuse: Transferring Omni Features With Pretrained Masked Autoencoders for Infrared and Visible Image Fusion via Guided TrainingTIP '25Paper/Code
TransformerFusion with Language-drivenInfrared and Visible Image Fusion with Language-Driven Loss in CLIP Embedding SpaceACM MM '24Paper/Code
\n\n## 数据兼容方法(Data Compatible)\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
Aspects
(分类)		Methods
(方法)		Title
(标题)		Venue
(发表场所)		Source
(资源)
RegistrationUMIRUnsupervised multi-modal image registration via geometry preserving image-to-image translationCVPR ‘20Paper/Code
RegistrationReCoNetReconet: Recurrent correction network for fast and efficient multi-modality image fusionECCV ‘22Paper/Code
RegistrationSuperFusionSuperfusion: A versatile image registration and fusion network with semantic awarenessJAS ‘22Paper/Code
RegistrationUMFusionUnsupervised misaligned infrared and visible image fusion via cross-modality image generation and registrationIJCAI ‘22Paper/Code
RegistrationGCRFGeneral cross-modality registration framework for visible and infrared UAV target image registrationSR ‘23Paper
RegistrationMURFMURF: mutually reinforcing multi-modal image registration and fusionTPAMI ‘23Paper/Code
RegistrationSemLASemantics lead all: Towards unified image registration and fusion from a semantic perspectiveInfFus ‘23Paper/Code
Registration-A Deep Learning Framework for Infrared and Visible Image Fusion Without Strict RegistrationIJCV ‘23Paper
AttackPAIFusionPAIF: Perception-aware infrared-visible image fusion for attack-tolerant semantic segmentationACMMM ‘23Paper/Code
GeneralFusionDNFusionDN: A unified densely connected network for image fusionAAAI ‘20Paper/Code
GeneralIFCNNIFCNN: A general image fusion framework based on convolutional neural networkInfFus ‘20Paper/Code
GeneralPMGIRethinking the image fusion: A fast unified image fusion network based on proportional maintenance of gradient and intensityAAAI ‘20Paper/Code
GeneralU2FusionU2Fusion: A unified unsupervised image fusion networkTPAMI ‘20Paper/Code
GeneralSDNetSDNet: A versatile squeeze-and-decomposition network for real-time image fusionIJCV ‘21Paper/Code
GeneralCoCoNetCoCoNet: Coupled contrastive learning network with multi-level feature ensemble for multi-modality image fusionIJCV ‘23Paper/Code
GeneralDDFMDDFM: Denoising diffusion model for multi-modality image fusionICCV ‘23Paper/Code
GeneralEMMAEquivariant multi-modality image fusionCVPR ‘24Paper/Code
GeneralFILMImage fusion via vision-language modelICML ‘24Paper/Code
GeneralVDMUFusionVDMUFusion: A Versatile Diffusion Model-Based Unsupervised Framework for Image FusionTIP ‘24Paper/Code
GeneralTC-MoATask-Customized Mixture of Adapters for General Image FusionCVPR '24Paper/Code
GeneralSHIPProbing Synergistic High-Order Interaction in Infrared and Visible Image FusionCVPR '24Paper/Code
TransformerGIFNetOne Model for ALL: Low-Level Task Interaction Is a Key to Task-Agnostic Image FusionCVPR '25Paper/Code
\n\n## 面向应用方法(Application-oriented)\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
Aspects
(分类)		Methods
(方法)		Title
(标题)		Venue
(发表场所)		Source
(资源)
PerceptionDetFusionA detection-driven infrared and visible image fusion networkACMMM ‘22Paper/Code
PerceptionSeAFusionImage fusion in the loop of high-level vision tasks: A semantic-aware real-time infrared and visible image fusion networkInfFus ‘22Paper/Code
PerceptionTarDALTarget-aware dual adversarial learning and a multi-scenario multimodality benchmark to fuse infrared and visible for object detectionCVPR ‘22Paper/Code
PerceptionBDLFusionBi-level dynamic learning for jointly multi-modality image fusion and beyondIJCAI ‘23Paper/Code
PerceptionIRFSAn interactively reinforced paradigm for joint infrared-visible image fusion and saliency object detectionInfFus ‘23Paper/Code
PerceptionMetaFusionMetafusion: Infrared and visible image fusion via meta-feature embedding from object detectionCVPR ‘23Paper/Code
PerceptionMoE-FusionMulti-modal gated mixture of local-to-global experts for dynamic image fusionICCV ‘23Paper/Code
PerceptionSegMiFMulti-interactive feature learning and a full-time multimodality benchmark for image fusion and segmentationICCV ‘23Paper/Code
PerceptionCAFWhere elegance meets precision: Towards a compact, automatic, and flexible framework for multi-modality image fusion and applicationsIJCAI ‘24Paper/Code
PerceptionMRFSMrfs: Mutually reinforcing image fusion and segmentationCVPR ‘24Paper/Code
PerceptionTIMFusionA task-guided, implicitly searched and meta-initialized deep model for image fusionTPAMI ‘24Paper/Code
PerceptionSAGEEvery SAM Drop Counts: Embracing Semantic Priors for Multi-Modality Image Fusion and BeyondCVPR ‘25Paper/Code
PerceptionDCEvoDCEvo: Discriminative Cross-Dimensional Evolutionary Learning for Infrared and Visible Image FusionCVPR ‘25Paper/Code
PerceptionTDFusionTask-driven Image Fusion with Learnable Fusion LossCVPR ‘25Paper/Code
PerceptionEVIFEvent-based Visible and Infrared Fusion via Multi-task Collaboration CVPR ‘24Paper/Code
PerceptionMMAIFMMAIF: Multi-task and Multi-degradation All-in-One for Image Fusion with Language GuidanceICCV ‘25Paper/Code
PerceptionCMFSCMFS: CLIP-Guided Modality Interaction for Mitigating Noise in Multi-Modal Image Fusion and SegmentationIJCAI ‘25Paper/Code
PerceptionA²RNetA²RNet: Adversarial Attack Resilient Network for Robust Infrared and Visible Image FusionAAAI ‘25Paper/Code
PerceptionSDSFusionSDSFusion: A Semantic-Aware Infrared and Visible Image Fusion Network for Degraded ScenesTIP ‘25Paper/Code
PerceptionS4FusionS4Fusion: Saliency-Aware Selective State Space Model for Infrared and Visible Image FusionTIP ‘25Paper/Code
PerceptionFreeFusionFreeFusion: Infrared and Visible Image Fusion via Cross Reconstruction LearningTPAMI ‘25Paper
PerceptionMulFS-CAPMulFS-CAP: Multimodal Fusion-Supervised Cross-Modality Alignment Perception for Unregistered Infrared-Visible Image FusionTPAMI ‘25Paper/Code
\n\n\n\n# 最新研究进展(Latest Research Progress)\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
Methods
(方法)		Title
(标题)		Venue
(发表场所)		Source
(资源)
SAGEEvery SAM Drop Counts: Embracing Semantic Priors for Multi-Modality Image Fusion and BeyondCVPR ‘25Paper/Code
DCEvoDCEvo: Discriminative Cross-Dimensional Evolutionary Learning for Infrared and Visible Image FusionCVPR ‘25Paper/Code
TDFusionTask-driven Image Fusion with Learnable Fusion LossCVPR ‘25Paper/Code
GIFNetOne Model for ALL: Low-Level Task Interaction Is a Key to Task-Agnostic Image FusionCVPR '25Paper/Code
RIS-FuseHighlight What You Want: Weakly-Supervised Instance-Level Controllable Infrared-Visible Image FusionICCV '25Paper/Code
SCAThe Source Image is the Best Attention for Infrared and Visible Image FusionICCV '25Paper
TITABalancing Task-invariant Interaction and Task-specific Adaptation for Unified Image FusionICCV '25Paper/Code
DreamFuseDreamFuse: Adaptive Image Fusion with Diffusion TransformerICCV '25Paper/Code
TRACEToward a Training-Free Plug-and-Play Refinement Framework for Infrared and Visible Image Registration and FusionACM MM '25Paper/Code
HRFusionPrior-Constrained Relevant Feature driven Image Fusion with Hybrid Feature via Mode DecompositionACM MM '25Paper/Code
TG-ECNetTask-Gated Multi-Expert Collaboration Network for Degraded Multi-Modal Image FusionICML '25Paper/Code
Deno-IFDeno-IF: Unsupervised Noisy Visible and Infrared Image Fusion via Mimicking Textures from Clean ImagesNeurIPS '25Paper/Code
HCLFuseRevisiting Generative Infrared and Visible Image Fusion Based on Human Cognitive LawsNeurIPS '25Paper/Code
RFfusionEfficient Rectified Flow for Image FusionNeurIPS '25Paper/Code
ControlFusionControlFusion: A Controllable Image Fusion Framework with Language-Vision Degradation PromptsNeurIPS '25Paper/Code
PDFuseProjection-Manifold Regularized Latent Diffusion for Robust General Image FusionNeurIPS '25Paper/Code
DAFusionDomain Adaptation Guided Infrared and Visible Image FusionAAAI '26Paper/Code
HMMFA Hybrid Space Model for Misaligned Multi-modality Image FusionAAAI '26Paper/Code
CtrlFuseCtrlFuse: Mask-Prompt Guided Controllable Infrared and Visible Image FusionAAAI '26Paper/Code
MdaIFMdaIF: Robust One-Stop Multi-Degradation-Aware Image Fusion with Language-Driven SemanticsAAAI '26Paper/Code
SMCSelf-supervised Multiplex Consensus Mamba for General Image FusionAAAI '26Paper
OCCOOCCO: LVM-Guided Infrared and Visible Image Fusion Framework Based on Object-Aware and Contextual Contrastive LearningIJCV '25Paper/Code
MagicFuseOne Model for ALL: Low-Level Task Interaction Is a Key to Task-Agnostic Image FusionCVPR '26Paper
UniFusionOne Model for ALL: Low-Level Task Interaction Is a Key to Task-Agnostic Image FusionCVPR '26Paper/Code
OmniFuseOmniFuse: Composite Degradation-Robust Image Fusion With Language-Driven SemanticsTPAMI '25Paper/Code
DiTFuseTowards Uniffed Semantic and Controllable Image Fusion: A Diffusion Transformer ApproachTPAMI '25Paper/Code
Mask-DiFuserMask-DiFuser: A Masked Diffusion Model for Unified Unsupervised Image FusionTPAMI '25Paper/Code
FuseAgentFUSEAGENT: A VLM-DRIVEN AGENT FOR UNIFIED IN-THE-WILD IMAGE FUSIONICLR '26Paper/Code
URFusionURFusion: Unsupervised Unified Degradation-Robust Image Fusion NetworkTIP '25Paper/Code
\n\n# 视频融合(Video Fusion)\n\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
Methods
(方法)		Title
(标题)		Venue
(发表场所)		Source
(资源)
UniVFA Unified Solution to Video Fusion: FromMulti-Frame Learning to BenchmarkingNeurIPS '25Paper/Code
RCVSRCVS: A Unified Registration and Fusion Framework for Video StreamsTMM '24Paper
VideoFusionVideoFusion: A Spatio-Temporal Collaborative Network for Multi-modal Video FusionCVPR '26Paper/Code
CMVFCMVF: Cross-modal unregistered video fusion via spatio-temporal consistencyInfFus '26Paper/Code
Seq-IFSeq-IF: Sequentially Consistent Infrared-Visible Video Fusion under Time-Varying Illumination for Perception EnhancementTCSVT '26Paper
TemCoCoTemCoCo: Temporally Consistent Multi-modal Video Fusion with Visual-Semantic CollaborationICCV '25Paper/Code
\n# 评价指标(Evaluation Metric)\nWe integrated the code for calculating metrics and used GPU acceleration with PyTorch, significantly improving the speed of computing metrics across multiple methods and images.\nYou can find it at [Metric](https://github.com/RollingPlain/IVIF_ZOO/tree/main/Metric)\n\nIf you want to calculate metrics using our code, you can run:\n```python\n# Please modify the data path in 'eval_torch.py'.\npython eval_torch.py\n ```\n\n# 资源库(Resource Library)\n## 融合(Fusion)\n\nFusion images from multiple datasets in the IVIF domain are organized in the following form: each subfolder contains fusion images generated by different methods, facilitating research and comparison for users.\n```\nFusion ROOT\n├── IVIF\n| ├── FMB\n| | ├── ... \n| | ├── CAF # All the file names are named after the methods\n| | └── ...\n| ├── # The other files follow the same structure shown above.\n| ├── M3FD_300 # Mini version of M3FD dataset with 300 images\n| ├── RoadScene\n| ├── TNO\n| └── M3FD_4200.zip # Full version of the M3FD dataset with 4200 images\n```\nYou can directly download from here.\n\nDownload：[Baidu Yun](https://pan.baidu.com/s/1S6l-CUqE2nRPXeX2P_VScg?pwd=wgtn)\n\n\n## 分割(Segmentation)\n\nSegmentation data is organized in the following form: it contains multiple directories to facilitate the management of segmentation-related data and results.\n\n```\nSegmentation ROOT\n├── Segformer\n| ├── datasets\n| | ├── ... \n| | ├── CAF # All the file names are named after the methods\n| | | └──VOC2007\n| | | ├── JPEGImages # Fusion result images in JPG format\n| | | └── SegmentationClass # Ground truth for segmentation\n| | └── ... # The other files follow the same structure shown above.\n| ├── model_data \n| | ├── backbone # Backbone used for segmentation\n| | └── model # Saved model files\n| | ├── ...\n| | ├── CAF.pth # All the model names are named after the methods\n| | └── ... \n| ├── results # Saved model files and training results\n| | ├── iou # IoU results for segmentation validation\n| | ├── ...\n| | ├── CAF.txt # All the file names are named after the methods\n| | └── ... \n| | └── predict #Visualization of segmentation\n| | ├── ...\n| | ├── CAF # All the file names are named after the methods\n| | └── ... \n| └── hyperparameters.md # Hyperparameter settings\n```\n\nYou can directly download from here.\n\nDownload：[Baidu Yun](https://pan.baidu.com/s/1IZOZU17CA6-zeR8zb1LW3Q?pwd=5rcp)\n\n## 检测(Detection)\nDetection data is organized in the following form:\nit contains multiple directories to facilitate the management of detection-related data and results.\n\n```\nDetection ROOT\n├── M3FD\n| ├── Fused Results\n| | ├── ... \n| | ├── CAF # All the file names are named after the methods\n| | | ├── Images # Fusion result images in PNG format\n| | | └── Labels # Ground truth for detection\n| | └── ... # The other files follow the same structure shown above.\n| ├── model_data \n| | └── model # Saved model files\n| | ├── ...\n| | ├── CAF.pth # All the model names are named after the methods\n| | └── ... \n| ├── results # Saved model files and training results\n| | └── predict #Visualization of detection\n| | ├── ...\n| | ├── CAF # All the file names are named after the methods\n| | └── ... \n| └── hyperparameters.md # Hyperparameter settings\n```\n\nYou can directly download from here.\n\nDownload：[Baidu Yun](https://pan.baidu.com/s/1mC3wTM1DjbBz5mIaDYJLDQ?pwd=a36k)\n## 计算效率(Computational Efficiency)\n- **FLOPS and Params**:\n - We utilize the `profile` function from the `thop` package to compute the FLOPs (G) and Params (M) counts of the model.\n```python\nfrom thop import profile\n\n# Create ir, vi input tensor\nir = torch.randn(1, 1, 1024, 768).to(device)\nvi = torch.randn(1, 3, 1024, 768).to(device)\n# Assume 'model' is your network model\nflops, params = profile(model, inputs=(ir, vi))\n ```\n\n- **Time**:\n - To measure the Time (ms) of the model, we exclude the initial image to compute the average while testing a random selection of 10 image sets from the M3FD dataset, each with a resolution of 1024×768, on the Nvidia GeForce 4090. To eliminate CPU influence, we employ CUDA official event functions to measure running time on the GPU.\n\n```python\nimport torch\n \n# Create CUDA events\nstart = torch.cuda.Event(enable_timing=True)\nend = torch.cuda.Event(enable_timing=True)\n# Record the start time\nstart.record()\n# Execute the model\n# Assume 'model' is your network model\nfus = model(ir, vi) \n# Record the end time\nend.record()\n```\n\n" } ]