FuzzBench: compare-sancov-3 report

(experiment incomplete/still running...)
warning
Please consider this as a preliminary report to demonstrate the capabilities of FuzzBench. While we have tried our best, we have not confirmed that we configured everything correctly. We are hoping to work together with the community to validate results and improve the set of fuzzers, benchmarks, and their configurations in the future. See FAQ for more details.

experiment summary

We show two different aggregate (cross-benchmark) rankings of fuzzers. The first is based on the average of per-benchmarks scores, where the score represents the percentage of the highest reached median coverage on a given benchmark (higher value is better). The second ranking shows the average rank of fuzzers, after we rank them on each benchmark according to their median reached covereges (lower value is better).
By avg. score
average normalized score
fuzzer
aflplusplus_optimal 96.82
honggfuzz 95.98
aflsmart 95.11
aflplusplus 94.85
afl 94.41
entropic 93.31
aflfast 92.24
fastcgs_lm 91.73
mopt 90.87
lafintel 88.58
aflcc 88.01
libfuzzer 87.03
fairfuzz 81.12
eclipser 68.40
manul 58.37
By avg. rank
average rank
fuzzer
aflplusplus_optimal 2.90
honggfuzz 3.62
afl 4.50
aflplusplus 4.97
fastcgs_lm 5.29
aflsmart 5.57
entropic 6.27
mopt 6.53
aflcc 7.42
aflfast 7.69
libfuzzer 7.94
fairfuzz 8.03
lafintel 8.52
eclipser 11.75
manul 12.40
  • Critical difference diagram
    The diagram visualizes the average rank of fuzzers (second ranking above) while showing the significance of the differences as well. What is considered a "critical difference" (CD) is based on the Friedman/Nemenyi post-hoc test. See more in the documentation.
    Note: If a fuzzer does not support all benchmarks, its ranking as shown in this diagram can be lower than it should be. So please check the list of supported benchmarks for the fuzzer(s) of your interest. The list could be specified in the fuzzer's README.md like this.
  • Median coverages on each benchmark
    fuzzer afl aflcc aflfast aflplusplus aflplusplus_optimal aflsmart eclipser entropic fairfuzz fastcgs_lm honggfuzz lafintel libfuzzer manul mopt
    benchmark
    bloaty_fuzz_target 5413.0 5203.0 5264.5 5324.0 5664.0 5444.0 4172.0 NaN 5296.0 5669.0 5523.0 5005.5 4505.0 3932.0 5654.0
    curl_curl_fuzzer_http 5424.5 5537.0 NaN 5421.0 5588.0 5424.0 4414.5 5399.5 4814.0 5414.0 5428.0 5313.0 4804.5 3511.0 5374.0
    freetype2-2017 5361.0 6440.0 5236.0 5260.0 6395.0 5274.0 4523.0 5730.0 5325.0 5333.0 7091.0 5126.0 4224.0 3899.0 5334.0
    harfbuzz-1.3.2 4265.0 4137.5 4157.0 4386.0 4356.0 4247.0 3366.5 4501.5 3650.0 4255.5 4382.5 4093.0 4034.0 2350.5 4241.0
    jsoncpp_jsoncpp_fuzzer 634.0 630.0 NaN 634.0 634.0 634.0 584.0 628.0 634.0 634.0 634.5 631.0 NaN 454.0 634.0
    lcms-2017-03-21 1091.5 844.5 NaN 991.5 1233.0 1082.5 453.5 1217.0 1017.5 869.0 1005.0 936.0 1103.0 419.0 842.0
    libjpeg-turbo-07-2017 NaN 1424.5 NaN 1429.0 1429.5 NaN 1036.0 NaN 1118.0 1435.0 NaN 1263.0 1380.5 1059.0 1434.0
    libpcap_fuzz_both NaN 1698.0 NaN NaN 1595.0 NaN 791.0 1503.0 121.0 NaN 1669.0 1378.0 1473.0 18.0 NaN
    libpng-1.2.56 628.5 542.0 626.0 627.5 627.0 672.5 478.0 NaN 629.0 520.0 668.0 638.0 611.5 510.5 521.0
    libxml2-v2.9.2 4642.5 3586.0 4411.0 4590.0 4720.0 4541.0 1669.5 4519.0 3397.5 3893.0 4564.0 4271.0 4328.0 1353.0 3832.5
    mbedtls_fuzz_dtlsclient 1656.0 979.0 NaN 1660.0 1669.0 1648.0 1361.0 1630.0 1658.5 1658.0 1650.0 1568.0 1611.0 1323.0 1655.0
    openssl_x509 NaN 4070.0 NaN 4074.0 4075.5 NaN 4050.0 NaN 4052.0 4074.0 4066.5 4064.5 4064.0 4045.0 4073.0
    openthread-2019-12-23 1463.0 1474.0 1643.0 1471.0 1657.5 NaN 1439.0 NaN 1091.0 1657.0 1646.5 1460.0 1478.0 1089.0 1466.5
    proj4-2017-08-14 NaN 2068.0 NaN NaN 2275.5 NaN 350.0 NaN 2024.0 1849.0 3212.0 2052.0 NaN 528.0 1947.0
    re2-2014-12-09 NaN 2225.0 2243.0 2249.0 2270.0 2245.0 1893.0 NaN 2243.0 2228.0 2264.0 2230.0 2280.0 854.0 2225.0
    sqlite3_ossfuzz 17242.0 14859.0 16718.0 17347.5 17239.0 17243.0 5406.0 13800.5 10231.5 17637.0 12461.0 11896.5 8758.5 8061.0 17146.0
    systemd_fuzz-link-parser NaN 778.0 NaN 985.5 997.0 NaN 921.0 NaN 938.0 987.0 NaN 976.0 981.0 864.0 987.0
    vorbis-2017-12-11 1011.5 997.0 NaN 1001.0 1000.0 1010.0 856.5 1003.0 970.0 1011.0 NaN 973.0 NaN 834.5 1009.0
    woff2-2016-05-06 1136.0 1138.0 NaN 1114.5 1177.0 1133.5 742.0 1108.5 NaN 1140.5 1205.0 1112.0 921.0 813.0 1139.0
    zlib_zlib_uncompress_fuzzer 331.0 328.0 NaN 330.0 328.0 323.5 307.0 NaN 329.0 327.0 NaN 327.0 334.0 209.0 328.0

bloaty_fuzz_target summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    fastcgs_lm 82800 18.0 5676.444444 55.888778 5590.0 5639.00 5669.0 5709.25 5797.0
    aflplusplus_optimal 82800 16.0 5660.000000 57.436342 5520.0 5620.00 5664.0 5699.75 5742.0
    mopt 82800 17.0 5630.882353 74.763696 5456.0 5581.00 5654.0 5689.00 5718.0
    honggfuzz 82800 17.0 5509.000000 55.278839 5362.0 5497.00 5523.0 5538.00 5585.0
    aflsmart 82800 18.0 5419.833333 157.343106 5141.0 5292.00 5444.0 5532.00 5680.0
    afl 82800 18.0 5375.333333 150.107413 5139.0 5258.25 5413.0 5460.00 5641.0
    aflplusplus 82800 15.0 5285.000000 125.442645 5066.0 5186.00 5324.0 5378.50 5468.0
    fairfuzz 82800 17.0 5262.235294 225.708753 4714.0 5203.00 5296.0 5425.00 5566.0
    aflfast 82800 14.0 5253.285714 153.725744 5054.0 5161.00 5264.5 5302.75 5640.0
    aflcc 82800 17.0 5200.058824 62.359312 5032.0 5188.00 5203.0 5237.00 5304.0
    lafintel 82800 18.0 5040.444444 132.660656 4880.0 4943.75 5005.5 5095.25 5338.0
    libfuzzer 82800 13.0 4485.076923 127.661963 4227.0 4385.00 4505.0 4579.00 4647.0
    eclipser 82800 17.0 4192.176471 79.494682 4041.0 4146.00 4172.0 4228.00 4330.0
    manul 82800 18.0 3930.166667 31.537559 3866.0 3911.00 3932.0 3954.75 3980.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

curl_curl_fuzzer_http summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus_optimal 82800 18.0 5587.666667 21.646899 5559.0 5571.25 5588.0 5600.25 5639.0
    aflcc 82800 18.0 5534.722222 31.739390 5462.0 5514.75 5537.0 5558.50 5587.0
    honggfuzz 82800 15.0 5390.800000 121.040843 4987.0 5369.50 5428.0 5441.00 5507.0
    afl 82800 6.0 5426.666667 35.764042 5381.0 5403.00 5424.5 5448.25 5478.0
    aflsmart 82800 10.0 5433.400000 36.539780 5394.0 5403.75 5424.0 5455.75 5493.0
    aflplusplus 82800 15.0 5429.866667 31.955028 5370.0 5416.00 5421.0 5435.50 5505.0
    fastcgs_lm 82800 7.0 5409.428571 16.811277 5377.0 5405.00 5414.0 5417.50 5430.0
    entropic 82800 6.0 5405.500000 21.257940 5381.0 5392.75 5399.5 5419.00 5437.0
    mopt 82800 8.0 5375.125000 18.247798 5342.0 5368.50 5374.0 5385.25 5398.0
    lafintel 82800 17.0 5308.823529 35.164320 5207.0 5295.00 5313.0 5327.00 5380.0
    fairfuzz 82800 15.0 4801.400000 319.895697 3777.0 4761.50 4814.0 4932.50 5198.0
    libfuzzer 82800 2.0 4804.500000 101.116270 4733.0 4768.75 4804.5 4840.25 4876.0
    eclipser 82800 18.0 4408.500000 40.379450 4337.0 4384.75 4414.5 4429.50 4481.0
    manul 82800 14.0 3518.357143 29.950233 3498.0 3499.25 3511.0 3520.75 3611.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

freetype2-2017 summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    honggfuzz 82800 19.0 7080.526316 246.467345 6674.0 6891.50 7091.0 7220.5 7703.0
    aflcc 82800 13.0 6467.153846 129.594140 6232.0 6363.00 6440.0 6570.0 6648.0
    aflplusplus_optimal 82800 13.0 6408.615385 256.795164 5813.0 6352.00 6395.0 6558.0 6775.0
    entropic 82800 19.0 5794.842105 219.666227 5561.0 5663.00 5730.0 5785.5 6219.0
    afl 82800 17.0 5356.882353 36.147065 5287.0 5343.00 5361.0 5392.0 5410.0
    mopt 82800 15.0 5339.533333 32.315115 5281.0 5319.50 5334.0 5356.0 5406.0
    fastcgs_lm 82800 17.0 5335.941176 33.347546 5252.0 5325.00 5333.0 5352.0 5386.0
    fairfuzz 82800 17.0 5331.705882 44.330809 5271.0 5296.00 5325.0 5363.0 5401.0
    aflsmart 82800 15.0 5280.266667 48.548160 5158.0 5259.50 5274.0 5316.0 5360.0
    aflplusplus 82800 9.0 5255.444444 46.481478 5174.0 5233.00 5260.0 5275.0 5325.0
    aflfast 82800 14.0 5239.642857 67.292253 5121.0 5198.25 5236.0 5283.0 5362.0
    lafintel 82800 17.0 5145.941176 127.939669 4955.0 5099.00 5126.0 5208.0 5542.0
    eclipser 82800 13.0 4473.846154 168.013812 4193.0 4370.00 4523.0 4553.0 4679.0
    libfuzzer 82800 17.0 4510.882353 500.440291 3852.0 4069.00 4224.0 5054.0 5178.0
    manul 82800 19.0 3852.631579 127.000967 3592.0 3775.50 3899.0 3932.5 4025.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

harfbuzz-1.3.2 summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    entropic 82800 12.0 4510.666667 58.621022 4438.0 4465.00 4501.5 4544.75 4616.0
    aflplusplus 82800 15.0 4364.000000 57.696249 4257.0 4344.00 4386.0 4399.00 4437.0
    honggfuzz 82800 18.0 4372.944444 43.508582 4227.0 4356.00 4382.5 4398.75 4427.0
    aflplusplus_optimal 82800 19.0 4332.526316 84.688559 4227.0 4255.50 4356.0 4403.50 4464.0
    afl 82800 13.0 4267.846154 45.403462 4217.0 4240.00 4265.0 4275.00 4404.0
    fastcgs_lm 82800 14.0 4254.928571 15.208622 4227.0 4243.00 4255.5 4262.75 4279.0
    aflsmart 82800 17.0 4261.470588 46.199456 4212.0 4239.00 4247.0 4256.00 4377.0
    mopt 82800 17.0 4262.882353 74.957390 4192.0 4215.00 4241.0 4253.00 4438.0
    aflfast 82800 17.0 4165.647059 42.250653 4113.0 4146.00 4157.0 4172.00 4293.0
    aflcc 82800 12.0 4161.833333 58.156268 4104.0 4127.75 4137.5 4183.00 4276.0
    lafintel 82800 17.0 4111.941176 50.212636 4063.0 4082.00 4093.0 4123.00 4231.0
    libfuzzer 82800 7.0 4040.285714 74.057892 3934.0 3991.50 4034.0 4096.00 4139.0
    fairfuzz 82800 17.0 3660.235294 197.557754 3322.0 3486.00 3650.0 3783.00 4039.0
    eclipser 82800 16.0 3364.937500 25.087763 3311.0 3351.75 3366.5 3376.50 3422.0
    manul 82800 16.0 2351.125000 32.190837 2312.0 2329.00 2350.5 2361.25 2433.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

jsoncpp_jsoncpp_fuzzer summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    honggfuzz 16200 2.0 634.500000 0.707107 634.0 634.25 634.5 634.75 635.0
    afl 16200 3.0 634.000000 0.000000 634.0 634.00 634.0 634.00 634.0
    aflplusplus 16200 1.0 634.000000 NaN 634.0 634.00 634.0 634.00 634.0
    aflplusplus_optimal 16200 20.0 634.000000 0.000000 634.0 634.00 634.0 634.00 634.0
    aflsmart 16200 1.0 634.000000 NaN 634.0 634.00 634.0 634.00 634.0
    fairfuzz 16200 18.0 633.777778 0.427793 633.0 634.00 634.0 634.00 634.0
    fastcgs_lm 16200 16.0 633.875000 0.341565 633.0 634.00 634.0 634.00 634.0
    mopt 16200 15.0 633.866667 0.351866 633.0 634.00 634.0 634.00 634.0
    lafintel 16200 18.0 631.166667 0.383482 631.0 631.00 631.0 631.00 632.0
    aflcc 16200 20.0 629.150000 2.680829 624.0 626.75 630.0 631.00 633.0
    entropic 16200 1.0 628.000000 NaN 628.0 628.00 628.0 628.00 628.0
    eclipser 16200 19.0 578.000000 15.275252 545.0 572.50 584.0 589.00 593.0
    manul 16200 20.0 456.350000 15.513237 418.0 448.50 454.0 464.50 486.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

lcms-2017-03-21 summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus_optimal 22500 15.0 1244.266667 43.758210 1176.0 1220.50 1233.0 1265.00 1341.0
    entropic 22500 2.0 1217.000000 77.781746 1162.0 1189.50 1217.0 1244.50 1272.0
    libfuzzer 22500 8.0 1113.750000 56.610827 1040.0 1072.00 1103.0 1152.50 1198.0
    afl 22500 8.0 1110.875000 71.066242 987.0 1083.50 1091.5 1174.00 1199.0
    aflsmart 22500 4.0 1077.250000 41.120757 1025.0 1055.75 1082.5 1104.00 1119.0
    fairfuzz 22500 18.0 1020.611111 105.676053 872.0 921.50 1017.5 1101.75 1250.0
    honggfuzz 22500 2.0 1005.000000 186.676190 873.0 939.00 1005.0 1071.00 1137.0
    aflplusplus 22500 2.0 991.500000 0.707107 991.0 991.25 991.5 991.75 992.0
    lafintel 22500 18.0 973.666667 87.173930 899.0 927.50 936.0 961.25 1235.0
    fastcgs_lm 22500 8.0 869.750000 30.127112 838.0 841.75 869.0 898.50 901.0
    aflcc 22500 18.0 847.388889 11.976147 835.0 841.00 844.5 848.75 880.0
    mopt 22500 11.0 851.181818 18.247291 839.0 842.00 842.0 852.50 903.0
    eclipser 22500 18.0 474.666667 80.047045 391.0 449.25 453.5 475.25 783.0
    manul 22500 19.0 415.736842 11.488362 395.0 410.50 419.0 424.50 432.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libjpeg-turbo-07-2017 summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    fastcgs_lm 48600 17.0 1430.117647 45.911440 1266.0 1431.00 1435.0 1451.00 1472.0
    mopt 48600 18.0 1416.333333 64.415928 1235.0 1431.50 1434.0 1445.25 1452.0
    aflplusplus_optimal 48600 16.0 1384.750000 77.841292 1265.0 1291.25 1429.5 1432.25 1466.0
    aflplusplus 48600 4.0 1389.500000 82.383251 1266.0 1386.75 1429.0 1431.75 1434.0
    aflcc 48600 18.0 1362.222222 84.859438 1235.0 1277.25 1424.5 1432.00 1447.0
    libfuzzer 48600 6.0 1374.166667 55.463201 1272.0 1368.75 1380.5 1412.50 1424.0
    lafintel 48600 16.0 1322.062500 83.410206 1256.0 1257.00 1263.0 1421.25 1442.0
    fairfuzz 48600 17.0 1100.647059 110.391882 910.0 1051.00 1118.0 1224.00 1225.0
    manul 48600 19.0 1018.947368 74.794142 897.0 945.50 1059.0 1077.00 1091.0
    eclipser 48600 16.0 1041.437500 49.230030 974.0 1018.50 1036.0 1054.50 1133.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libpcap_fuzz_both summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflcc 25200 18.0 1690.444444 105.964046 1500.0 1642.50 1698.0 1754.00 1875.0
    honggfuzz 25200 15.0 1682.333333 188.355919 1123.0 1620.50 1669.0 1780.50 1903.0
    aflplusplus_optimal 25200 15.0 1569.000000 222.376129 1103.0 1421.50 1595.0 1690.00 1940.0
    entropic 25200 8.0 1450.500000 180.315596 1165.0 1392.00 1503.0 1554.75 1672.0
    libfuzzer 25200 20.0 1502.550000 62.081843 1422.0 1460.50 1473.0 1548.00 1647.0
    lafintel 25200 20.0 1393.150000 116.994501 1125.0 1325.75 1378.0 1432.00 1695.0
    eclipser 25200 17.0 800.294118 79.773715 681.0 754.00 791.0 830.00 1028.0
    fairfuzz 25200 17.0 83.588235 71.170621 18.0 21.00 121.0 124.00 271.0
    manul 25200 19.0 62.000000 53.250978 17.0 18.00 18.0 121.00 126.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libpng-1.2.56 summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflsmart 13500 2.0 672.500000 10.606602 665.0 668.75 672.5 676.25 680.0
    honggfuzz 13500 16.0 668.000000 5.452828 659.0 664.75 668.0 670.00 679.0
    lafintel 13500 20.0 638.550000 1.791060 636.0 637.00 638.0 640.00 642.0
    fairfuzz 13500 20.0 628.600000 1.500877 626.0 627.75 629.0 630.00 631.0
    afl 13500 2.0 628.500000 2.121320 627.0 627.75 628.5 629.25 630.0
    aflplusplus 13500 2.0 627.500000 0.707107 627.0 627.25 627.5 627.75 628.0
    aflplusplus_optimal 13500 1.0 627.000000 NaN 627.0 627.00 627.0 627.00 627.0
    aflfast 13500 1.0 626.000000 NaN 626.0 626.00 626.0 626.00 626.0
    libfuzzer 13500 2.0 611.500000 10.606602 604.0 607.75 611.5 615.25 619.0
    aflcc 13500 20.0 540.150000 6.393047 530.0 534.00 542.0 543.00 555.0
    mopt 13500 17.0 520.941176 2.771971 517.0 519.00 521.0 523.00 525.0
    fastcgs_lm 13500 15.0 519.800000 2.426049 517.0 518.00 520.0 521.00 524.0
    manul 13500 20.0 508.450000 9.896703 489.0 503.00 510.5 515.00 522.0
    eclipser 13500 19.0 479.578947 12.816291 448.0 471.50 478.0 489.50 498.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libxml2-v2.9.2 summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus_optimal 82800 16.0 4715.625000 82.604984 4552.0 4688.25 4720.0 4760.75 4866.0
    afl 82800 18.0 4614.833333 118.369437 4280.0 4559.50 4642.5 4695.75 4755.0
    aflplusplus 82800 15.0 4542.600000 139.524089 4295.0 4412.50 4590.0 4667.00 4714.0
    honggfuzz 82800 17.0 4548.352941 83.762269 4395.0 4505.00 4564.0 4598.00 4703.0
    aflsmart 82800 17.0 4578.058824 103.739741 4445.0 4488.00 4541.0 4695.00 4739.0
    entropic 82800 1.0 4519.000000 NaN 4519.0 4519.00 4519.0 4519.00 4519.0
    aflfast 82800 15.0 4421.133333 114.703884 4157.0 4394.00 4411.0 4470.00 4623.0
    libfuzzer 82800 13.0 4351.307692 139.175180 4181.0 4242.00 4328.0 4444.00 4627.0
    lafintel 82800 19.0 4254.263158 149.868402 4036.0 4116.00 4271.0 4380.50 4474.0
    fastcgs_lm 82800 20.0 3802.050000 405.424595 2184.0 3782.25 3893.0 3994.50 4171.0
    mopt 82800 18.0 3713.888889 429.486819 2150.0 3662.25 3832.5 3928.75 4060.0
    aflcc 82800 17.0 3652.823529 311.090066 2967.0 3485.00 3586.0 3795.00 4186.0
    fairfuzz 82800 18.0 3493.388889 257.415923 3186.0 3352.00 3397.5 3537.00 4006.0
    eclipser 82800 16.0 1664.375000 25.080869 1616.0 1655.75 1669.5 1679.00 1714.0
    manul 82800 17.0 1345.529412 85.936108 1217.0 1265.00 1353.0 1397.00 1516.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

mbedtls_fuzz_dtlsclient summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus_optimal 38700 18.0 1670.388889 11.453943 1655.0 1662.00 1669.0 1677.25 1693.0
    aflplusplus 38700 6.0 1656.833333 12.480652 1638.0 1649.75 1660.0 1662.75 1673.0
    fairfuzz 38700 16.0 1657.500000 11.616081 1638.0 1651.25 1658.5 1665.00 1678.0
    fastcgs_lm 38700 15.0 1671.800000 70.284321 1622.0 1649.00 1658.0 1662.50 1922.0
    afl 38700 3.0 1655.333333 8.020806 1647.0 1651.50 1656.0 1659.50 1663.0
    mopt 38700 11.0 1652.727273 13.777452 1632.0 1645.00 1655.0 1660.50 1679.0
    honggfuzz 38700 9.0 1653.000000 11.672618 1636.0 1647.00 1650.0 1661.00 1670.0
    aflsmart 38700 3.0 1650.000000 14.106736 1637.0 1642.50 1648.0 1656.50 1665.0
    entropic 38700 1.0 1630.000000 NaN 1630.0 1630.00 1630.0 1630.00 1630.0
    libfuzzer 38700 1.0 1611.000000 NaN 1611.0 1611.00 1611.0 1611.00 1611.0
    lafintel 38700 17.0 1565.058824 6.259698 1547.0 1561.00 1568.0 1570.00 1572.0
    eclipser 38700 17.0 1368.411765 16.620991 1350.0 1358.00 1361.0 1375.00 1408.0
    manul 38700 19.0 1322.421053 8.871401 1308.0 1317.00 1323.0 1325.50 1347.0
    aflcc 38700 18.0 978.444444 0.704792 977.0 978.00 979.0 979.00 979.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

openssl_x509 summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus_optimal 19800 12.0 4074.500000 3.060006 4066.0 4074.50 4075.5 4076.00 4077.0
    aflplusplus 19800 1.0 4074.000000 NaN 4074.0 4074.00 4074.0 4074.00 4074.0
    fastcgs_lm 19800 20.0 4074.350000 1.631112 4071.0 4073.00 4074.0 4075.25 4078.0
    mopt 19800 17.0 4072.647059 2.119587 4069.0 4071.00 4073.0 4074.00 4075.0
    aflcc 19800 20.0 4069.850000 3.437793 4064.0 4067.50 4070.0 4073.25 4075.0
    honggfuzz 19800 8.0 4066.250000 2.251983 4063.0 4064.75 4066.5 4067.25 4070.0
    lafintel 19800 20.0 4064.800000 2.839570 4059.0 4063.00 4064.5 4067.00 4071.0
    libfuzzer 19800 2.0 4064.000000 1.414214 4063.0 4063.50 4064.0 4064.50 4065.0
    fairfuzz 19800 15.0 4052.400000 6.674044 4044.0 4046.00 4052.0 4055.00 4066.0
    eclipser 19800 17.0 4050.588235 2.829727 4047.0 4048.00 4050.0 4053.00 4056.0
    manul 19800 18.0 4044.777778 0.808452 4044.0 4044.00 4045.0 4045.00 4046.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

openthread-2019-12-23 summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus_optimal 18900 16.0 1657.437500 8.876702 1635.0 1655.75 1657.5 1659.25 1682.0
    fastcgs_lm 18900 7.0 1577.000000 102.283919 1466.0 1468.50 1657.0 1659.00 1661.0
    honggfuzz 18900 10.0 1620.400000 58.621574 1461.0 1617.75 1646.5 1651.00 1654.0
    aflfast 18900 1.0 1643.000000 NaN 1643.0 1643.00 1643.0 1643.00 1643.0
    libfuzzer 18900 5.0 1475.200000 9.757049 1464.0 1466.00 1478.0 1482.00 1486.0
    aflcc 18900 18.0 1532.277778 85.035498 1457.0 1463.25 1474.0 1631.25 1645.0
    aflplusplus 18900 7.0 1525.142857 99.887317 1460.0 1465.50 1471.0 1567.00 1680.0
    mopt 18900 8.0 1532.250000 98.165676 1452.0 1461.50 1466.5 1640.00 1661.0
    afl 18900 3.0 1526.666667 110.273901 1463.0 1463.00 1463.0 1558.50 1654.0
    lafintel 18900 13.0 1459.769231 5.614770 1450.0 1457.00 1460.0 1464.00 1468.0
    eclipser 18900 19.0 1464.894737 56.391385 1430.0 1435.50 1439.0 1445.00 1602.0
    fairfuzz 18900 20.0 1244.900000 201.380995 1091.0 1091.00 1091.0 1442.75 1620.0
    manul 18900 19.0 1089.000000 0.000000 1089.0 1089.00 1089.0 1089.00 1089.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

proj4-2017-08-14 summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    honggfuzz 49500 14.0 3205.642857 54.890951 3091.0 3186.25 3212.0 3247.25 3277.0
    aflplusplus_optimal 49500 18.0 2295.444444 115.070425 2101.0 2198.00 2275.5 2369.75 2563.0
    aflcc 49500 20.0 2093.950000 98.173035 1946.0 2036.50 2068.0 2144.00 2316.0
    lafintel 49500 11.0 2060.363636 33.197207 2022.0 2038.00 2052.0 2077.50 2134.0
    fairfuzz 49500 19.0 2004.210526 57.890298 1862.0 2002.00 2024.0 2044.50 2069.0
    mopt 49500 13.0 1935.846154 53.516113 1823.0 1883.00 1947.0 1974.00 2001.0
    fastcgs_lm 49500 17.0 1837.529412 133.984196 1543.0 1775.00 1849.0 1947.00 2020.0
    manul 49500 16.0 513.125000 97.715147 358.0 437.00 528.0 579.25 699.0
    eclipser 49500 19.0 390.105263 142.189582 276.0 306.50 350.0 382.50 805.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

re2-2014-12-09 summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libfuzzer 36900 11.0 2274.090909 13.772832 2254.0 2259.50 2280.0 2283.50 2293.0
    aflplusplus_optimal 36900 3.0 2272.000000 10.148892 2263.0 2266.50 2270.0 2276.50 2283.0
    honggfuzz 36900 17.0 2265.470588 6.568082 2256.0 2262.00 2264.0 2270.00 2279.0
    aflplusplus 36900 3.0 2250.000000 4.582576 2246.0 2247.50 2249.0 2252.00 2255.0
    aflsmart 36900 1.0 2245.000000 NaN 2245.0 2245.00 2245.0 2245.00 2245.0
    aflfast 36900 5.0 2242.800000 1.788854 2241.0 2241.00 2243.0 2244.00 2245.0
    fairfuzz 36900 18.0 2237.500000 28.029921 2127.0 2240.25 2243.0 2246.00 2256.0
    lafintel 36900 18.0 2228.666667 6.525966 2218.0 2223.00 2230.0 2233.75 2237.0
    fastcgs_lm 36900 15.0 2227.466667 4.926121 2220.0 2223.00 2228.0 2230.50 2235.0
    aflcc 36900 17.0 2224.823529 8.812174 2211.0 2220.00 2225.0 2233.00 2238.0
    mopt 36900 9.0 2225.000000 3.240370 2221.0 2223.00 2225.0 2226.00 2230.0
    eclipser 36900 15.0 1892.266667 14.444310 1865.0 1882.00 1893.0 1901.00 1919.0
    manul 36900 17.0 850.352941 35.395870 771.0 838.00 854.0 874.00 899.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

sqlite3_ossfuzz summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    fastcgs_lm 82800 17.0 17644.235294 178.054953 17359.0 17525.00 17637.0 17762.00 17966.0
    aflplusplus 82800 16.0 17308.562500 175.065690 16998.0 17199.75 17347.5 17408.75 17641.0
    aflsmart 82800 15.0 17244.400000 273.341440 16876.0 17027.00 17243.0 17396.00 17823.0
    afl 82800 17.0 17252.647059 216.329304 16880.0 17099.00 17242.0 17434.00 17637.0
    aflplusplus_optimal 82800 17.0 17257.294118 276.757286 16674.0 17121.00 17239.0 17499.00 17772.0
    mopt 82800 17.0 17103.235294 200.777405 16775.0 16925.00 17146.0 17240.00 17353.0
    aflfast 82800 16.0 16686.000000 209.016108 16178.0 16587.50 16718.0 16787.25 17008.0
    aflcc 82800 11.0 14954.090909 338.698230 14459.0 14729.00 14859.0 15284.00 15435.0
    entropic 82800 18.0 13766.444444 487.928300 12807.0 13563.75 13800.5 14128.50 14398.0
    honggfuzz 82800 16.0 12433.625000 429.880894 11728.0 12175.50 12461.0 12654.50 13414.0
    lafintel 82800 18.0 11814.722222 543.844247 10355.0 11560.50 11896.5 12160.25 12606.0
    fairfuzz 82800 10.0 10573.300000 1975.825231 8490.0 8702.25 10231.5 12415.50 13307.0
    libfuzzer 82800 8.0 8717.375000 282.067336 8238.0 8584.00 8758.5 8947.00 9020.0
    manul 82800 17.0 8086.647059 222.525825 7703.0 7953.00 8061.0 8218.00 8563.0
    eclipser 82800 15.0 5794.400000 673.379939 5400.0 5402.50 5406.0 6069.50 7111.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

vorbis-2017-12-11 summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 37800 6.0 1011.000000 8.148620 1000.0 1005.25 1011.5 1017.00 1021.0
    fastcgs_lm 37800 18.0 1012.000000 7.227399 1000.0 1007.25 1011.0 1014.00 1029.0
    aflsmart 37800 13.0 1015.923077 12.822256 1002.0 1008.00 1010.0 1024.00 1043.0
    mopt 37800 17.0 1009.117647 4.846163 1000.0 1007.00 1009.0 1011.00 1018.0
    entropic 37800 1.0 1003.000000 NaN 1003.0 1003.00 1003.0 1003.00 1003.0
    aflplusplus 37800 9.0 1002.777778 4.944132 998.0 999.00 1001.0 1007.00 1011.0
    aflplusplus_optimal 37800 17.0 1000.411765 5.601995 993.0 996.00 1000.0 1001.00 1015.0
    aflcc 37800 18.0 997.055556 3.207935 992.0 995.00 997.0 998.75 1006.0
    lafintel 37800 19.0 971.631579 8.447291 949.0 969.50 973.0 976.50 984.0
    fairfuzz 37800 1.0 970.000000 NaN 970.0 970.00 970.0 970.00 970.0
    eclipser 37800 18.0 831.444444 59.917809 669.0 808.50 856.5 860.25 904.0
    manul 37800 18.0 835.166667 11.152684 818.0 826.75 834.5 843.00 853.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

woff2-2016-05-06 summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    honggfuzz 27000 13.0 1204.384615 39.986536 1141.0 1176.00 1205.0 1243.00 1255.0
    aflplusplus_optimal 27000 11.0 1169.090909 35.169460 1098.0 1149.50 1177.0 1195.50 1203.0
    fastcgs_lm 27000 12.0 1140.333333 17.343237 1113.0 1128.50 1140.5 1150.75 1172.0
    mopt 27000 12.0 1152.083333 40.100914 1100.0 1122.50 1139.0 1179.50 1219.0
    aflcc 27000 19.0 1150.789474 34.785979 1107.0 1123.50 1138.0 1180.50 1238.0
    afl 27000 7.0 1138.571429 14.751917 1125.0 1127.00 1136.0 1147.50 1160.0
    aflsmart 27000 6.0 1144.833333 31.339538 1113.0 1126.00 1133.5 1160.50 1196.0
    aflplusplus 27000 10.0 1128.700000 34.438512 1098.0 1105.25 1114.5 1136.50 1204.0
    lafintel 27000 19.0 1113.842105 13.841416 1093.0 1102.50 1112.0 1126.50 1139.0
    entropic 27000 2.0 1108.500000 2.121320 1107.0 1107.75 1108.5 1109.25 1110.0
    libfuzzer 27000 7.0 929.000000 22.832725 903.0 917.50 921.0 935.00 974.0
    manul 27000 19.0 819.789474 10.298754 810.0 811.00 813.0 830.50 835.0
    eclipser 27000 11.0 736.272727 31.818519 681.0 736.00 742.0 748.00 795.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

zlib_zlib_uncompress_fuzzer summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libfuzzer 10800 2.0 334.000000 0.000000 334.0 334.00 334.0 334.00 334.0
    afl 10800 1.0 331.000000 NaN 331.0 331.00 331.0 331.00 331.0
    aflplusplus 10800 3.0 329.333333 2.081666 327.0 328.50 330.0 330.50 331.0
    fairfuzz 10800 20.0 329.750000 3.024027 324.0 329.00 329.0 331.25 337.0
    aflcc 10800 18.0 327.833333 3.417085 320.0 327.00 328.0 328.75 335.0
    aflplusplus_optimal 10800 17.0 328.294118 2.543735 324.0 327.00 328.0 329.00 333.0
    mopt 10800 14.0 327.000000 3.063432 318.0 327.00 328.0 329.00 329.0
    fastcgs_lm 10800 17.0 326.823529 2.505875 323.0 324.00 327.0 329.00 330.0
    lafintel 10800 18.0 327.666667 3.360672 320.0 325.25 327.0 329.00 334.0
    aflsmart 10800 2.0 323.500000 0.707107 323.0 323.25 323.5 323.75 324.0
    eclipser 10800 20.0 307.550000 3.776311 299.0 305.75 307.0 308.50 315.0
    manul 10800 20.0 207.950000 8.976431 185.0 206.00 209.0 211.25 229.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

experiment data

You can download the raw data for this report here.

Check out the documentation on how to create customized reports using this data. Also see some example Colab notebooks for doing custom analysis on the data here.

The experiment was conducted using this FuzzBench commit: a36b6e50b631a8287601e5822985db9caf31845b