本文介绍了基于图像结构相关性的经典质量评估方法SSIM和MS-SSIM。SSIM通过亮度、对比度、结构三个模块计算图像相似度；MS-SSIM则是多尺度的SSIM。文中还提及Paddle的实现，包括paddle_msssim包的使用，测试对比显示其与其他实现结果接近且速度有优势，并给出了计算指标和作为损失函数的示例。

引入

• 自然图像具有极高的结构性，表现在图像的像素间存在着很强的相关性，尤其是在空间相似的情况下，这些相关性在视觉场景中携带着关于物体结构的重要信息
• 我们假设人类视觉系统主要从可视区域内获取结构信息，人类视觉系统所以通过探测结构信息是否改变来感知图像失真的近似信息
• 大多数的基于误差敏感度（Error Sensitivity）的质量评估方法（如 MSE, PSNR）使用线性变换来分解图像信号，这不会涉及到相关性
• 而接下来要介绍的 SSIM 和 MS-SSIM 方法，便是通过图像结构相关性来衡量图像之间相似度和评估图像质量的两种经典的方法

参考资料

• SSIM 研究主页

• SSIM 论文

• MS-SSIM 论文

• Paddle-MSSSIM

算法介绍

SSIM

• SSIM（Structural SIMilarity）即结构相似性指数，是一种测量两个图像之间相似性的方法

• 假定其中一幅图像具有完美的质量，则 SSIM 指数可以被视为另一幅图像质量的度量。

• SSIM 指数的计算流程如下图所示：

  

• 由 SSIM 测量系统可得相似度的测量可由三种对比模块组成，分别为：亮度（l），对比度（c），结构（s），各个模块的计算公式如下：

  

• 总体的计算公式如下：

  

MS-SSIM

• MS-SSIM（Multi-Scale Structural Similarity）即多尺度结构相似性指数

• 是一种基于多尺度（图片按照一定规则，由大到小缩放）的 SSIM 指数

• 具体的计算公式如下：

  

Paddle 实现

• 基于?Pytorch MS-SSIM?项目开发了一个快速、可微分的 SSIM 和 MS-SSIM 的 Paddle 实现

• 可以通过安装并调用 paddle_msssim 包快速实现 SSIM 和 MS-SSIM 的计算

• Paddle MS-SSIM 与 SKImage、TensorFlow 和 Pytorch MS-SSIM 实现的测试对比结果如下：

  outputs(AMD Ryzen 4600H): =================================== Test SSIM=======================================> Single ImageRepeat 10 timessigma=0.0 ssim_skimage=1.000000 (247.7732 ms), ssim_tf=1.000000 (277.2696 ms), ssim_paddle=1.000000 (179.4677 ms), ssim_torch=1.000000 (183.6994 ms)sigma=10.0 ssim_skimage=0.932399 (226.1620 ms), ssim_tf=0.932640 (257.2435 ms), ssim_paddle=0.932636 (163.2263 ms), ssim_torch=0.932400 (179.1418 ms)sigma=20.0 ssim_skimage=0.786023 (224.1826 ms), ssim_tf=0.786032 (279.2126 ms), ssim_paddle=0.786017 (158.3070 ms), ssim_torch=0.786027 (180.0890 ms)sigma=30.0 ssim_skimage=0.637174 (237.5582 ms), ssim_tf=0.637183 (267.6092 ms), ssim_paddle=0.637165 (167.9277 ms), ssim_torch=0.637178 (181.7910 ms)sigma=40.0 ssim_skimage=0.515865 (221.0388 ms), ssim_tf=0.515876 (264.3230 ms), ssim_paddle=0.515857 (170.7676 ms), ssim_torch=0.515869 (189.0941 ms)sigma=50.0 ssim_skimage=0.422551 (222.6846 ms), ssim_tf=0.422558 (273.1971 ms), ssim_paddle=0.422542 (168.3579 ms), ssim_torch=0.422554 (176.7442 ms)sigma=60.0 ssim_skimage=0.351337 (215.1536 ms), ssim_tf=0.351340 (270.5560 ms), ssim_paddle=0.351325 (164.3315 ms), ssim_torch=0.351340 (194.6781 ms)sigma=70.0 ssim_skimage=0.295752 (210.0273 ms), ssim_tf=0.295756 (272.1814 ms), ssim_paddle=0.295744 (169.3864 ms), ssim_torch=0.295755 (178.9230 ms)sigma=80.0 ssim_skimage=0.253164 (239.2978 ms), ssim_tf=0.253169 (260.8894 ms), ssim_paddle=0.253157 (184.7061 ms), ssim_torch=0.253166 (181.4640 ms)sigma=90.0 ssim_skimage=0.219240 (224.7329 ms), ssim_tf=0.219245 (270.3727 ms), ssim_paddle=0.219235 (172.3580 ms), ssim_torch=0.219242 (180.5838 ms)sigma=100.0 ssim_skimage=0.192630 (238.8582 ms), ssim_tf=0.192634 (261.4317 ms), ssim_paddle=0.192624 (166.0294 ms), ssim_torch=0.192632 (175.7241 ms)Pass!====> BatchPass!登录后复制

  =================================== Test MS-SSIM=======================================> Single ImageRepeat 10 timessigma=0.0 msssim_tf=1.000000 (534.9398 ms), msssim_paddle=1.000000 (231.7381 ms), msssim_torch=1.000000 (257.3238 ms)sigma=10.0 msssim_tf=0.991148 (525.1758 ms), msssim_paddle=0.991147 (213.8527 ms), msssim_torch=0.991101 (243.9299 ms)sigma=20.0 msssim_tf=0.967450 (523.3070 ms), msssim_paddle=0.967447 (217.2415 ms), msssim_torch=0.967441 (253.1073 ms)sigma=30.0 msssim_tf=0.934692 (538.5145 ms), msssim_paddle=0.934687 (215.2203 ms), msssim_torch=0.934692 (242.5429 ms)sigma=40.0 msssim_tf=0.897363 (558.0346 ms), msssim_paddle=0.897357 (219.1107 ms), msssim_torch=0.897362 (249.1027 ms)sigma=50.0 msssim_tf=0.859276 (524.8582 ms), msssim_paddle=0.859267 (232.4189 ms), msssim_torch=0.859275 (263.1328 ms)sigma=60.0 msssim_tf=0.820967 (512.8726 ms), msssim_paddle=0.820958 (223.7422 ms), msssim_torch=0.820965 (251.9713 ms)sigma=70.0 msssim_tf=0.784204 (529.6149 ms), msssim_paddle=0.784194 (213.1742 ms), msssim_torch=0.784203 (244.9676 ms)sigma=80.0 msssim_tf=0.748574 (545.3014 ms), msssim_paddle=0.748563 (222.8581 ms), msssim_torch=0.748572 (261.0413 ms)sigma=90.0 msssim_tf=0.715980 (538.3886 ms), msssim_paddle=0.715968 (214.4464 ms), msssim_torch=0.715977 (282.6247 ms)sigma=100.0 msssim_tf=0.683882 (540.9150 ms), msssim_paddle=0.683870 (218.5596 ms), msssim_torch=0.683880 (244.1856 ms)Pass====> BatchPass登录后复制

• 具体的安装使用方法如下：

安装

!pip install paddle_msssim登录后复制

计算 SSIM 和 MS-SSIM 指标

• 这里使用如下三张图像来计算他们之间的 SSIM 和 MS-SSIM 指标，结果如下：

  Image@@##@@@@##@@@@##@@Simga050100SSIM1.0000000.4229270.192567MS-SSIM1.0000000.8588610.684299

• 具体的计算代码如下：

import cv2import paddlefrom paddle_msssim import ssim, ms_ssimdef imread(img_path): img = cv2.imread(img_path) return paddle.to_tensor(img.transpose(2, 0, 1)[None, ...], dtype=paddle.float32)simga_0 = imread('./images/simga_0.png')simga_50 = imread('./images/simga_50.png')simga_100 = imread('./images/simga_100.png')ssim_0 = ssim(simga_0, simga_0)ssim_50 = ssim(simga_0, simga_50)ssim_100 = ssim(simga_0, simga_100)print('[SSIM] simga_0: %f simga_50: %f simga_100: %f' % (ssim_0, ssim_50, ssim_100))ms_ssim_0 = ms_ssim(simga_0, simga_0)ms_ssim_50 = ms_ssim(simga_0, simga_50)ms_ssim_100 = ms_ssim(simga_0, simga_100)print('[MS-SSIM] simga_0: %f simga_50: %f simga_100: %f' % (ms_ssim_0, ms_ssim_50, ms_ssim_100))登录后复制

[SSIM] simga_0: 1.000000 simga_50: 0.422927 simga_100: 0.192567[MS-SSIM] simga_0: 1.000000 simga_50: 0.858861 simga_100: 0.684299登录后复制

作为损失函数使用

• 随机初始化的一张雪花图像，使用 SSIM 和 MS-SSIM 作为损失函数去拟合目标图像

import osimport sysimport paddleimport numpy as npfrom PIL import Imagefrom paddle.optimizer import Adamfrom paddle_msssim import SSIM, MS_SSIMloss_type = 'ssim'assert loss_type in ['ssim', 'msssim']if loss_type == 'ssim': loss_obj = SSIM(win_size=11, win_sigma=1.5, data_range=1, size_average=True, channel=3)else: loss_obj = MS_SSIM(win_size=11, win_sigma=1.5, data_range=1, size_average=True, channel=3)np_img1 = np.array(Image.open(”./images/simga_0.png“))img1 = paddle.to_tensor(np_img1.transpose(2, 0 , 1)).unsqueeze(0) / 255.0img2 = paddle.rand(img1.shape)img1 = paddle.to_tensor(img1, stop_gradient=True)img2 = paddle.to_tensor(img2, stop_gradient=False)with paddle.no_grad(): ssim_value = loss_obj(img1, img2).item() print(”Initial %s: %f:“ % (loss_type, ssim_value))optimizer = Adam(parameters=[img2], learning_rate=0.05)step = 0while ssim_value < 0.9999: step += 1 optimizer.clear_grad() loss = loss_obj(img1, img2) (1 - loss).backward() optimizer.step() ssim_value = loss.item() if step % 10 == 0: print('step: %d %s: %f' % (step, loss_type, ssim_value))img2_ = (img2 * 255.0).squeeze()np_img2 = img2_.detach().numpy().astype(np.uint8).transpose(1, 2, 0)results = Image.fromarray(np.concatenate([np_img1, np_img2], 1))results.save('results_%s.png' % loss_type)results登录后复制

Initial ssim: 0.010401:step: 10 ssim: 0.225660step: 20 ssim: 0.733606step: 30 ssim: 0.919254step: 40 ssim: 0.970057step: 50 ssim: 0.990348step: 60 ssim: 0.998122step: 70 ssim: 0.999767登录后复制

<PIL.Image.Image image mode=RGB size=1024x768 at 0x7F157A7B6F50>登录后复制

更多

• 更多使用细节和示例可以参考 Paddle-MSSSIM 的?Github 仓库