FuzzBench: 2024-08-25-2038-libaf report

experiment summary

We show two different aggregate (cross-benchmark) rankings of fuzzers. The first is based on the average of per-benchmarks scores, where the score represents the percentage of the highest reached median code-coverage on a given benchmark (higher value is better). The second ranking shows the average rank of fuzzers, after we rank them on each benchmark according to their median reached code-covereges (lower value is better).
By avg. score
average normalized score
fuzzer
aflplusplus 96.83
mopt 82.97
eclipser 82.67
aflsmart 82.60
aflfast 81.93
libafl 80.52
fairfuzz 79.26
centipede 70.79
By avg. rank
average rank
fuzzer
aflplusplus 1.71
libafl 3.38
aflsmart 4.00
eclipser 4.19
mopt 4.76
fairfuzz 5.52
centipede 5.67
aflfast 5.81
  • Critical difference diagram
    The diagram visualizes the average rank of fuzzers (second ranking above) while showing the significance of the differences as well. What is considered a "critical difference" (CD) is based on the Friedman/Nemenyi post-hoc test. See more in the documentation.
    Note: If a fuzzer does not support all benchmarks, its ranking as shown in this diagram can be lower than it should be. So please check the list of supported benchmarks for the fuzzer(s) of your interest. The list could be specified in the fuzzer's README.md like this.
  • Median relative code-coverages on each benchmark

    Note: The relative coverage summary table shows the median relative performance of each fuzzer to the experiment maximum. Thus the highest relative performance may not be 100%.
    trial_relative_coverage = trial_coverage / experiment_max_coverage

      libafl aflplusplus eclipser mopt aflsmart aflfast centipede fairfuzz
    FuzzerMedian 98.00 98.00 94.00 93.00 94.00 93.00 91.00 85.00
    FuzzerMean 96.00 93.38 87.89 80.33 80.00 79.48 79.33 76.86
    bloaty_fuzz_target 99.00 98.00 95.00 96.00 95.00 95.00 nan 81.00
    curl_curl_fuzzer_http 98.00 98.00 93.00 93.00 94.00 93.00 nan 84.00
    freetype2_ftfuzzer 90.00 91.00 73.00 66.00 66.00 63.00 56.00 62.00
    harfbuzz_hb-shape-fuzzer nan 98.00 97.00 97.00 97.00 96.00 nan 85.00
    jsoncpp_jsoncpp_fuzzer 98.00 99.00 98.00 98.00 98.00 98.00 98.00 98.00
    lcms_cms_transform_fuzzer 94.00 93.00 77.00 52.00 41.00 29.00 36.00 56.00
    libjpeg-turbo_libjpeg_turbo_fuzzer 99.00 82.00 nan 82.00 82.00 98.00 97.00 99.00
    libpcap_fuzz_both nan 88.00 71.00 0.00 0.00 1.00 81.00 0.00
    libpng_libpng_read_fuzzer 98.00 98.00 97.00 96.00 97.00 95.00 98.00 97.00
    libxml2_xml nan 99.00 97.00 97.00 97.00 96.00 93.00 89.00
    libxslt_xpath 96.00 98.00 94.00 93.00 95.00 93.00 94.00 95.00
    mbedtls_fuzz_dtlsclient 92.00 72.00 70.00 70.00 70.00 68.00 69.00 73.00
    openssl_x509 99.00 99.00 99.00 99.00 99.00 99.00 99.00 99.00
    openthread_ot-ip6-send-fuzzer 84.00 72.00 70.00 68.00 68.00 68.00 67.00 65.00
    proj4_proj_crs_to_crs_fuzzer 96.00 85.00 62.00 10.00 10.00 9.00 10.00 10.00
    re2_fuzzer 99.00 99.00 99.00 99.00 99.00 99.00 96.00 99.00
    sqlite3_ossfuzz nan 98.00 93.00 93.00 93.00 93.00 63.00 59.00
    systemd_fuzz-link-parser 99.00 100.00 91.00 91.00 92.00 92.00 98.00 87.00
    vorbis_decode_fuzzer 98.00 99.00 98.00 98.00 98.00 98.00 89.00 97.00
    woff2_convert_woff2ttf_fuzzer 98.00 97.00 nan 93.00 93.00 91.00 89.00 82.00
    zlib_zlib_uncompress_fuzzer 95.00 98.00 96.00 96.00 96.00 95.00 95.00 97.00
    • Fuzzers are sorted by "FuzzerMean" (average median relative coverage), highest on the left.
    • Green background = highest relative median coverage.
    • Blue gradient background = greater than 95% relative median coverage.

bloaty_fuzz_target summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
error
The following fuzzers do not have enough samples: eclipser.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libafl 82800 16.0 6314.312500 53.836442 6198.0 6282.25 6328.5 6364.00 6372.0
    aflplusplus 82800 19.0 6285.789474 36.073350 6220.0 6261.50 6290.0 6308.50 6370.0
    mopt 82800 20.0 6189.050000 93.155487 6019.0 6121.00 6188.0 6260.50 6323.0
    aflsmart 82800 20.0 6146.000000 146.262561 5784.0 6070.75 6104.0 6271.00 6381.0
    eclipser 82800 11.0 6053.181818 68.930136 5914.0 6022.50 6078.0 6095.50 6135.0
    aflfast 82800 20.0 6077.150000 116.111233 5846.0 6016.00 6076.0 6140.25 6335.0
    fairfuzz 82800 16.0 5189.937500 67.159977 5111.0 5135.25 5170.0 5226.00 5323.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

curl_curl_fuzzer_http summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libafl 82800 20.0 10904.350000 83.178486 10697.0 10852.00 10905.5 10963.50 11038.0
    aflplusplus 82800 18.0 10873.888889 101.307853 10660.0 10797.50 10883.0 10936.50 11057.0
    aflsmart 82800 20.0 10385.000000 132.985951 10018.0 10363.25 10420.0 10455.25 10544.0
    eclipser 82800 17.0 10351.470588 163.547057 9942.0 10353.00 10381.0 10436.00 10545.0
    mopt 82800 20.0 10355.100000 76.350232 10128.0 10334.25 10353.0 10406.75 10458.0
    aflfast 82800 20.0 10300.900000 57.898187 10100.0 10277.00 10306.0 10340.75 10375.0
    fairfuzz 82800 19.0 9222.578947 391.462545 8215.0 8926.50 9379.0 9512.00 9714.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

freetype2_ftfuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus 82800 18.0 11435.000000 371.083469 10320.0 11329.25 11492.5 11674.25 11975.0
    libafl 82800 19.0 11525.210526 618.204261 10585.0 11081.50 11364.0 11891.00 12620.0
    eclipser 82800 18.0 9312.222222 98.567717 9066.0 9283.00 9317.0 9355.50 9516.0
    aflsmart 82800 20.0 8342.750000 174.588501 7870.0 8314.75 8393.0 8435.50 8535.0
    mopt 82800 20.0 8321.650000 153.060454 7821.0 8324.75 8357.5 8392.00 8480.0
    aflfast 82800 20.0 8059.000000 238.403903 7700.0 7852.50 8061.5 8289.00 8362.0
    fairfuzz 82800 17.0 7936.117647 236.373349 7764.0 7809.00 7863.0 7895.00 8582.0
    centipede 82800 20.0 7221.200000 196.544250 6889.0 7115.75 7184.0 7243.50 7661.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

harfbuzz_hb-shape-fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
error
The following fuzzers do not have enough samples: fairfuzz.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus 82800 20.0 10909.550000 56.417126 10835.0 10855.25 10897.5 10954.75 11029.0
    mopt 82800 20.0 10778.950000 52.249880 10651.0 10757.50 10789.5 10796.25 10863.0
    eclipser 82800 20.0 10769.400000 51.530676 10623.0 10744.25 10777.0 10789.50 10868.0
    aflsmart 82800 20.0 10763.700000 42.521945 10652.0 10753.00 10764.5 10791.00 10839.0
    aflfast 82800 20.0 10677.750000 59.487172 10566.0 10633.00 10675.5 10714.50 10791.0
    fairfuzz 82800 14.0 9509.571429 347.646530 9071.0 9259.50 9399.0 9715.75 10155.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

jsoncpp_jsoncpp_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus 82800 20.0 519.800000 0.410391 519.0 520.00 520.0 520.00 520.0
    centipede 82800 20.0 519.850000 2.007224 518.0 518.00 519.0 522.00 524.0
    eclipser 82800 20.0 516.200000 5.176872 505.0 516.50 518.0 520.00 520.0
    mopt 82800 20.0 517.050000 4.430457 502.0 517.00 518.0 520.00 520.0
    aflfast 82800 20.0 517.000000 3.670652 502.0 517.00 517.0 519.00 519.0
    aflsmart 82800 19.0 516.105263 4.689169 503.0 517.00 517.0 519.00 520.0
    fairfuzz 82800 16.0 516.437500 2.988171 509.0 516.75 517.0 518.25 520.0
    libafl 82800 19.0 517.210526 0.917663 516.0 517.00 517.0 517.00 519.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

lcms_cms_transform_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libafl 82800 16.0 2013.687500 140.240136 1558.0 1995.00 2035.5 2079.75 2159.0
    aflplusplus 82800 20.0 1843.750000 259.035026 1409.0 1558.75 2011.0 2067.25 2117.0
    eclipser 82800 17.0 1656.823529 137.802955 1398.0 1537.00 1671.0 1753.00 1893.0
    fairfuzz 82800 18.0 1274.277778 397.407600 800.0 901.00 1221.0 1636.25 1940.0
    mopt 82800 20.0 1154.600000 428.242602 650.0 697.50 1133.0 1559.00 1736.0
    aflsmart 82800 20.0 1074.750000 469.254714 650.0 652.00 900.0 1561.00 1791.0
    centipede 82800 20.0 938.450000 266.317277 732.0 778.50 796.0 949.50 1469.0
    aflfast 82800 20.0 655.900000 152.566086 476.0 629.50 640.5 644.25 1276.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

libjpeg-turbo_libjpeg_turbo_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
error
The following fuzzers do not have enough samples: fairfuzz.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    fairfuzz 82800 13.0 3072.615385 17.399897 3017.0 3072.00 3079.0 3080.00 3084.0
    libafl 82800 16.0 3079.187500 1.973787 3076.0 3078.00 3079.0 3080.25 3083.0
    aflfast 82800 20.0 3047.050000 30.163545 3007.0 3015.00 3053.5 3078.00 3081.0
    centipede 82800 20.0 3011.300000 37.937761 2962.0 2977.75 3003.5 3058.50 3066.0
    aflplusplus 82800 18.0 2547.333333 2.029199 2545.0 2545.25 2547.0 2548.00 2552.0
    aflsmart 82800 20.0 2544.750000 1.773341 2543.0 2543.00 2545.0 2546.00 2549.0
    mopt 82800 20.0 2544.200000 1.542384 2540.0 2543.75 2544.0 2545.00 2547.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

libpcap_fuzz_both summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
error
The following fuzzers do not have enough samples: mopt, aflplusplus, aflfast.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus 82800 9.0 3043.222222 63.088782 2950.0 2991.0 3058.0 3091.0 3112.0
    centipede 82800 20.0 2413.350000 1012.985960 100.0 2591.5 2824.0 2926.5 3133.0
    eclipser 82800 19.0 2461.052632 196.678947 1977.0 2403.0 2446.0 2609.0 2706.0
    aflfast 82800 6.0 38.500000 4.929503 34.0 34.0 38.5 43.0 43.0
    aflsmart 82800 20.0 34.000000 0.000000 34.0 34.0 34.0 34.0 34.0
    fairfuzz 82800 16.0 37.375000 4.500000 34.0 34.0 34.0 43.0 43.0
    mopt 82800 14.0 36.214286 3.826599 34.0 34.0 34.0 37.0 43.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

libpng_libpng_read_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    centipede 82800 20.0 2015.300000 3.213459 2009.0 2013.00 2015.5 2017.00 2023.0
    aflplusplus 82800 19.0 2003.736842 2.765705 1999.0 2002.00 2003.0 2005.00 2011.0
    libafl 82800 18.0 1994.722222 9.176732 1977.0 1996.25 1998.0 2000.00 2004.0
    eclipser 82800 18.0 1983.000000 27.563830 1900.0 1988.25 1993.5 1996.25 1999.0
    aflsmart 82800 20.0 1964.400000 41.639303 1888.0 1926.50 1992.5 1995.25 1998.0
    fairfuzz 82800 18.0 1972.166667 30.020091 1889.0 1955.75 1981.0 1996.00 1999.0
    mopt 82800 20.0 1969.650000 25.915399 1915.0 1945.50 1976.5 1990.75 1999.0
    aflfast 82800 20.0 1940.050000 40.045336 1856.0 1926.25 1946.5 1973.50 1987.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

libxml2_xml summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus 82800 20.0 15654.900000 58.256149 15529.0 15627.00 15644.5 15684.75 15795.0
    eclipser 82800 17.0 15366.176471 70.190130 15247.0 15310.00 15373.0 15407.00 15477.0
    aflsmart 82800 20.0 15356.900000 64.464515 15142.0 15341.50 15371.0 15390.50 15431.0
    mopt 82800 20.0 15336.600000 74.989403 15154.0 15311.25 15343.0 15360.00 15510.0
    aflfast 82800 20.0 15282.950000 77.642551 15097.0 15249.50 15282.5 15340.75 15388.0
    centipede 82800 20.0 14733.700000 151.201539 14403.0 14652.50 14736.5 14818.00 15097.0
    fairfuzz 82800 19.0 14034.842105 437.228934 12545.0 13923.00 14065.0 14210.00 14843.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

libxslt_xpath summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
error
The following fuzzers do not have enough samples: libafl.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus 82800 18.0 11229.277778 89.675759 11102.0 11148.25 11226.0 11308.25 11375.0
    libafl 82800 12.0 11019.666667 44.593789 10947.0 10998.25 11019.0 11043.50 11107.0
    fairfuzz 82800 17.0 10804.470588 200.035346 10162.0 10772.00 10855.0 10906.00 11044.0
    aflsmart 82800 20.0 10844.850000 43.367797 10793.0 10816.50 10826.5 10859.25 10942.0
    eclipser 82800 18.0 10767.611111 123.770252 10424.0 10773.25 10801.5 10834.00 10888.0
    centipede 82800 20.0 10704.000000 102.787466 10557.0 10632.50 10700.0 10760.50 10912.0
    aflfast 82800 20.0 10631.400000 116.451662 10368.0 10570.25 10677.0 10705.50 10796.0
    mopt 82800 20.0 10575.400000 126.064896 10279.0 10519.00 10613.0 10662.00 10772.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

mbedtls_fuzz_dtlsclient summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
error
The following fuzzers do not have enough samples: fairfuzz, libafl.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libafl 82800 12.0 3399.583333 346.971693 2760.0 3265.25 3515.0 3643.50 3816.0
    fairfuzz 82800 15.0 2849.466667 159.218119 2754.0 2779.00 2801.0 2807.50 3296.0
    aflplusplus 82800 19.0 2805.578947 175.754031 2726.0 2737.50 2761.0 2795.00 3518.0
    aflsmart 82800 20.0 2709.000000 26.942434 2665.0 2701.75 2705.5 2713.00 2792.0
    eclipser 82800 16.0 2726.562500 280.074030 2496.0 2674.00 2692.5 2715.00 3730.0
    mopt 82800 20.0 2685.000000 34.546688 2562.0 2672.50 2692.5 2701.25 2726.0
    centipede 82800 20.0 2646.050000 22.818448 2613.0 2625.50 2641.0 2668.50 2677.0
    aflfast 82800 20.0 2561.850000 115.700737 2312.0 2580.00 2611.0 2629.75 2658.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

openssl_x509 summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
error
The following fuzzers do not have enough samples: eclipser, libafl.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus 82800 17.0 5834.823529 1.776066 5832.0 5833.00 5835.0 5836.0 5838.0
    libafl 82800 14.0 5829.428571 3.501962 5821.0 5830.00 5830.5 5831.0 5833.0
    aflsmart 82800 20.0 5826.300000 5.582869 5808.0 5827.00 5828.0 5829.0 5831.0
    eclipser 82800 15.0 5825.266667 4.096456 5817.0 5823.50 5827.0 5828.0 5831.0
    centipede 82800 20.0 5823.300000 6.966839 5811.0 5819.00 5826.5 5829.0 5831.0
    mopt 82800 20.0 5823.350000 6.507283 5813.0 5815.75 5826.0 5829.0 5830.0
    aflfast 82800 20.0 5822.100000 7.496666 5797.0 5820.00 5823.5 5826.5 5830.0
    fairfuzz 82800 16.0 5821.125000 2.753785 5816.0 5819.50 5822.0 5823.0 5825.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

openthread_ot-ip6-send-fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
error
The following fuzzers do not have enough samples: libafl.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libafl 82800 11.0 3517.090909 356.781293 3053.0 3306.50 3561.0 3579.00 4225.0
    aflplusplus 82800 19.0 3262.947368 242.101557 3046.0 3063.50 3075.0 3518.00 3599.0
    eclipser 82800 16.0 2984.937500 66.946216 2895.0 2918.25 2999.5 3036.75 3073.0
    mopt 82800 20.0 2889.450000 41.043590 2826.0 2832.75 2912.5 2916.00 2936.0
    aflsmart 82800 20.0 2896.650000 46.006035 2828.0 2886.25 2907.0 2912.25 3025.0
    aflfast 82800 20.0 2888.250000 43.931616 2810.0 2865.75 2906.0 2911.75 2974.0
    centipede 82800 20.0 2842.250000 60.918302 2742.0 2786.50 2868.0 2887.00 2956.0
    fairfuzz 82800 19.0 2779.105263 65.278799 2676.0 2745.50 2764.0 2801.50 2912.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

proj4_proj_crs_to_crs_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
error
The following fuzzers do not have enough samples: libafl.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libafl 82800 15.0 7476.066667 115.402071 7302.0 7411.00 7457.0 7523.50 7762.0
    aflplusplus 82800 20.0 6724.250000 169.893426 6460.0 6607.25 6672.0 6847.50 7029.0
    eclipser 82800 20.0 4850.350000 238.740175 4471.0 4688.75 4879.5 4962.25 5520.0
    centipede 82800 20.0 821.300000 4.461531 814.0 817.75 820.0 826.00 828.0
    fairfuzz 82800 20.0 779.350000 98.588606 494.0 810.00 816.0 819.00 821.0
    aflsmart 82800 20.0 805.450000 17.101785 740.0 804.75 807.0 815.25 821.0
    mopt 82800 20.0 798.400000 22.272003 740.0 798.50 806.5 808.00 820.0
    aflfast 82800 20.0 738.800000 92.916828 488.0 727.75 740.5 806.25 817.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

re2_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
error
The following fuzzers do not have enough samples: libafl.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflsmart 82800 20.0 2863.850000 17.502707 2792.0 2863.75 2867.5 2870.50 2876.0
    eclipser 82800 17.0 2849.176471 37.480387 2746.0 2854.00 2867.0 2872.00 2877.0
    aflplusplus 82800 19.0 2868.000000 5.185450 2861.0 2864.00 2866.0 2872.00 2878.0
    fairfuzz 82800 16.0 2849.875000 34.960692 2757.0 2858.75 2864.0 2866.00 2872.0
    aflfast 82800 20.0 2851.500000 30.130418 2778.0 2860.50 2862.5 2866.25 2869.0
    mopt 82800 20.0 2846.850000 31.551587 2764.0 2847.00 2861.5 2864.25 2868.0
    libafl 82800 12.0 2856.000000 6.980492 2844.0 2851.75 2857.0 2862.25 2864.0
    centipede 82800 20.0 2771.400000 17.333873 2737.0 2762.75 2771.5 2777.25 2802.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

sqlite3_ossfuzz summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
error
The following fuzzers do not have enough samples: fairfuzz.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus 82800 20.0 19686.100000 1201.247198 14717.0 19858.50 19963.5 20142.75 20304.0
    aflsmart 82800 20.0 18982.900000 237.661613 18306.0 18933.75 19022.0 19117.00 19303.0
    mopt 82800 20.0 18882.300000 305.529067 18331.0 18657.50 18968.0 19091.50 19473.0
    aflfast 82800 20.0 18918.450000 267.101494 18518.0 18677.25 18938.0 19154.00 19247.0
    eclipser 82800 16.0 18884.375000 348.908751 17767.0 18835.25 18909.5 19083.75 19306.0
    centipede 82800 20.0 12999.100000 482.476495 12195.0 12749.75 12917.0 13277.75 13947.0
    fairfuzz 82800 12.0 12516.333333 1555.113169 10873.0 11305.75 12095.0 13230.75 15546.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

vorbis_decode_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
error
The following fuzzers do not have enough samples: eclipser, libafl.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus 82800 20.0 1263.400000 2.835861 1259.0 1261.75 1263.0 1265.00 1270.0
    libafl 82800 5.0 1253.000000 3.082207 1250.0 1250.00 1253.0 1255.00 1257.0
    mopt 82800 20.0 1250.400000 9.626717 1218.0 1252.00 1253.0 1254.00 1256.0
    aflsmart 82800 20.0 1244.100000 19.558011 1199.0 1247.00 1251.5 1254.00 1259.0
    aflfast 82800 20.0 1245.400000 19.491969 1183.0 1246.75 1251.0 1253.75 1258.0
    eclipser 82800 15.0 1248.133333 5.514483 1236.0 1245.00 1248.0 1252.50 1255.0
    fairfuzz 82800 18.0 1228.111111 28.060625 1175.0 1208.25 1238.0 1250.75 1257.0
    centipede 82800 20.0 1146.000000 17.161769 1117.0 1134.00 1142.5 1159.00 1177.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

woff2_convert_woff2ttf_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
error
The following fuzzers do not have enough samples: libafl, aflsmart.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libafl 82800 14.0 1175.857143 14.627988 1143.0 1170.25 1177.0 1185.75 1198.0
    aflplusplus 82800 20.0 1160.000000 13.118890 1131.0 1150.50 1164.5 1171.50 1179.0
    aflsmart 82800 13.0 1105.461538 29.809438 1047.0 1090.00 1123.0 1127.00 1129.0
    mopt 82800 20.0 1115.450000 18.259749 1068.0 1106.25 1119.5 1128.75 1140.0
    aflfast 82800 20.0 1084.450000 26.198584 1032.0 1065.25 1090.5 1106.25 1124.0
    centipede 82800 20.0 1076.850000 11.370669 1062.0 1067.50 1077.5 1085.25 1099.0
    fairfuzz 82800 17.0 994.941176 30.150602 959.0 982.00 991.0 1001.00 1098.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

zlib_zlib_uncompress_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
error
The following fuzzers do not have enough samples: aflsmart.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus 82800 19.0 462.473684 4.426792 456.0 460.00 462.0 463.00 471.0
    fairfuzz 82800 16.0 457.875000 3.685557 455.0 455.75 457.0 459.00 470.0
    aflsmart 82800 12.0 455.750000 13.948770 416.0 455.75 456.0 462.00 470.0
    mopt 82800 20.0 456.100000 4.024922 449.0 455.00 455.5 458.50 464.0
    eclipser 82800 16.0 451.687500 13.189484 423.0 450.25 455.0 458.00 471.0
    centipede 82800 20.0 454.650000 6.499190 451.0 451.00 451.0 455.25 471.0
    libafl 82800 16.0 450.375000 4.112988 444.0 448.75 450.0 451.00 460.0
    aflfast 82800 20.0 448.100000 15.109774 386.0 449.00 449.0 454.00 460.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

experiment data

You can download the raw data for this report here.

Check out the documentation on how to create customized reports using this data. Also see some example Colab notebooks for doing custom analysis on the data here.

Experiment Description:

(None,)