FuzzBench: 2023-12-04-sanitizers-1 report

(experiment incomplete/still running...)

experiment summary

We show two different aggregate (cross-benchmark) rankings of fuzzers. The first is based on the average of per-benchmarks scores, where the score represents the percentage of the highest reached median code-coverage on a given benchmark (higher value is better). The second ranking shows the average rank of fuzzers, after we rank them on each benchmark according to their median reached code-covereges (lower value is better).
By avg. score
average normalized score
fuzzer
aflplusplus_afluse_asan 98.43
aflplusplus_sanflags_asan 93.93
aflplusplus_afluse_ubsan 83.36
aflplusplus_afluse_msan 68.22
By avg. rank
average rank
fuzzer
aflplusplus_afluse_asan 2.09
aflplusplus_afluse_ubsan 2.14
aflplusplus_sanflags_asan 2.45
aflplusplus_afluse_msan 3.05
  • Critical difference diagram
    The diagram visualizes the average rank of fuzzers (second ranking above) while showing the significance of the differences as well. What is considered a "critical difference" (CD) is based on the Friedman/Nemenyi post-hoc test. See more in the documentation.
    Note: If a fuzzer does not support all benchmarks, its ranking as shown in this diagram can be lower than it should be. So please check the list of supported benchmarks for the fuzzer(s) of your interest. The list could be specified in the fuzzer's README.md like this.
  • Median relative code-coverages on each benchmark

    Note: The relative coverage summary table shows the median relative performance of each fuzzer to the experiment maximum. Thus the highest relative performance may not be 100%.
    trial_relative_coverage = trial_coverage / experiment_max_coverage

      aflplusplus_afluse_asan aflplusplus_sanflags_asan aflplusplus_afluse_ubsan aflplusplus_afluse_msan
    FuzzerMedian 97.00 97.00 97.00 97.00
    FuzzerMean 93.64 93.48 92.26 89.50
    bloaty_fuzz_target 94.00 nan 97.00 nan
    curl_curl_fuzzer_http 98.00 99.00 nan nan
    freetype2_ftfuzzer 58.00 58.00 26.00 26.00
    harfbuzz_hb-shape-fuzzer 99.00 98.00 99.00 99.00
    jsoncpp_jsoncpp_fuzzer 100.00 99.00 100.00 nan
    lcms_cms_transform_fuzzer 81.00 87.00 93.00 74.00
    libjpeg-turbo_libjpeg_turbo_fuzzer 99.00 99.00 99.00 99.00
    libpcap_fuzz_both 90.00 88.00 89.00 92.00
    libpng_libpng_read_fuzzer 97.00 97.00 97.00 97.00
    libxml2_xml 98.00 98.00 99.00 99.00
    libxslt_xpath 98.00 98.00 98.00 99.00
    mbedtls_fuzz_dtlsclient 96.00 97.00 97.00 nan
    openssl_x509 99.00 99.00 99.00 99.00
    openthread_ot-ip6-send-fuzzer 84.00 84.00 83.00 83.00
    proj4_proj_crs_to_crs_fuzzer 90.00 89.00 96.00 nan
    re2_fuzzer 99.00 99.00 99.00 nan
    sqlite3_ossfuzz 95.00 90.00 90.00 79.00
    stb_stbi_read_fuzzer 97.00 97.00 96.00 96.00
    systemd_fuzz-link-parser 94.00 94.00 99.00 97.00
    vorbis_decode_fuzzer 99.00 99.00 nan 99.00
    woff2_convert_woff2ttf_fuzzer 98.00 97.00 nan 97.00
    zlib_zlib_uncompress_fuzzer 97.00 97.00 97.00 97.00
    • Fuzzers are sorted by "FuzzerMean" (average median relative coverage), highest on the left.
    • Green background = highest relative median coverage.
    • Blue gradient background = greater than 95% relative median coverage.

bloaty_fuzz_target summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus_afluse_ubsan 63900 6.0 6265.666667 74.188049 6196.0 6207.0 6242.0 6322.00 6370.0
    aflplusplus_afluse_asan 63900 6.0 6055.666667 127.134050 5884.0 5976.5 6050.0 6139.25 6229.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

curl_curl_fuzzer_http summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus_sanflags_asan 65700 7.0 10840.000 87.871118 10700.0 10791.5 10838.0 10908.0 10943.0
    aflplusplus_afluse_asan 65700 8.0 10817.375 40.376576 10779.0 10787.5 10810.5 10828.5 10903.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

freetype2_ftfuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus_afluse_asan 900 5.0 6407.400000 278.927051 6163.0 6251.0 6302.0 6451.0 6870.0
    aflplusplus_sanflags_asan 900 9.0 6334.666667 221.427753 6131.0 6221.0 6295.0 6320.0 6876.0
    aflplusplus_afluse_ubsan 900 10.0 2834.600000 0.516398 2834.0 2834.0 2835.0 2835.0 2835.0
    aflplusplus_afluse_msan 900 8.0 2834.500000 0.534522 2834.0 2834.0 2834.5 2835.0 2835.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

harfbuzz_hb-shape-fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus_afluse_ubsan 65700 2.0 10923.5 50.204581 10888.0 10905.75 10923.5 10941.25 10959.0
    aflplusplus_afluse_asan 65700 4.0 10887.5 49.088356 10827.0 10859.25 10893.0 10921.25 10937.0
    aflplusplus_afluse_msan 65700 3.0 10866.0 59.016947 10803.0 10839.00 10875.0 10897.50 10920.0
    aflplusplus_sanflags_asan 65700 2.0 10820.5 19.091883 10807.0 10813.75 10820.5 10827.25 10834.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

jsoncpp_jsoncpp_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus_afluse_asan 65700 7.0 520.0 0.000000 520.0 520.0 520.0 520.0 520.0
    aflplusplus_afluse_ubsan 65700 7.0 520.0 0.000000 520.0 520.0 520.0 520.0 520.0
    aflplusplus_sanflags_asan 65700 3.0 517.0 2.645751 514.0 516.0 518.0 518.5 519.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

lcms_cms_transform_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus_afluse_ubsan 65700 4.0 1931.25 265.003616 1539.0 1899.75 2032.5 2064.00 2121.0
    aflplusplus_sanflags_asan 65700 4.0 1849.25 238.263684 1519.0 1759.75 1908.5 1998.00 2061.0
    aflplusplus_afluse_asan 65700 4.0 1770.00 227.527288 1542.0 1592.25 1772.0 1949.75 1994.0
    aflplusplus_afluse_msan 65700 5.0 1790.60 318.112087 1506.0 1551.00 1629.0 2083.00 2184.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libjpeg-turbo_libjpeg_turbo_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus_afluse_ubsan 65700 6.0 2549.500000 2.428992 2547.0 2548.0 2548.5 2551.25 2553.0
    aflplusplus_afluse_msan 65700 4.0 2548.000000 1.632993 2546.0 2547.5 2548.0 2548.50 2550.0
    aflplusplus_afluse_asan 65700 2.0 2547.000000 1.414214 2546.0 2546.5 2547.0 2547.50 2548.0
    aflplusplus_sanflags_asan 65700 3.0 2547.333333 0.577350 2547.0 2547.0 2547.0 2547.50 2548.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libpcap_fuzz_both summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus_afluse_msan 65700 3.0 3034.333333 170.907382 2901.0 2938.00 2975.0 3101.00 3227.0
    aflplusplus_afluse_asan 65700 2.0 2917.500000 13.435029 2908.0 2912.75 2917.5 2922.25 2927.0
    aflplusplus_afluse_ubsan 65700 8.0 2865.375000 173.729624 2612.0 2780.50 2891.5 2987.50 3092.0
    aflplusplus_sanflags_asan 65700 7.0 2875.857143 111.108698 2758.0 2808.50 2849.0 2921.00 3065.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libpng_libpng_read_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus_afluse_asan 65700 6.0 2012.666667 22.703891 1995.0 2003.50 2005.5 2008.25 2058.0
    aflplusplus_sanflags_asan 65700 4.0 2004.000000 1.414214 2003.0 2003.00 2003.5 2004.50 2006.0
    aflplusplus_afluse_ubsan 65700 2.0 2001.500000 0.707107 2001.0 2001.25 2001.5 2001.75 2002.0
    aflplusplus_afluse_msan 65700 7.0 2000.714286 3.093773 1996.0 1999.00 2000.0 2003.50 2004.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libxml2_xml summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus_afluse_ubsan 65700 7.0 15701.285714 57.267543 15604.0 15669.50 15717.0 15740.00 15769.0
    aflplusplus_afluse_msan 65700 4.0 15609.250000 70.320101 15505.0 15598.75 15638.0 15648.50 15656.0
    aflplusplus_afluse_asan 65700 4.0 15506.750000 21.868928 15480.0 15495.00 15508.0 15519.75 15531.0
    aflplusplus_sanflags_asan 65700 10.0 15493.200000 62.770835 15363.0 15457.75 15493.5 15544.00 15565.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libxslt_xpath summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus_afluse_msan 65700 8.0 11240.125 61.557029 11142.0 11207.5 11254.5 11271.0 11328.0
    aflplusplus_afluse_ubsan 65700 8.0 11232.625 69.038162 11162.0 11184.0 11197.5 11293.0 11342.0
    aflplusplus_sanflags_asan 65700 7.0 11181.000 64.059868 11104.0 11122.0 11195.0 11238.0 11248.0
    aflplusplus_afluse_asan 65700 3.0 11145.000 81.061705 11070.0 11102.0 11134.0 11182.5 11231.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

mbedtls_fuzz_dtlsclient summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus_afluse_ubsan 65700 4.0 2783.750000 62.590601 2704.0 2760.25 2787.5 2811.00 2856.0
    aflplusplus_sanflags_asan 65700 12.0 2779.833333 31.653905 2724.0 2770.50 2779.0 2793.75 2845.0
    aflplusplus_afluse_asan 65700 8.0 2761.500000 38.541258 2720.0 2725.75 2761.0 2793.00 2813.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

openssl_x509 summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus_afluse_asan 65700 9.0 5822.555556 5.637178 5813.0 5819.0 5825.0 5827.0 5829.0
    aflplusplus_sanflags_asan 65700 8.0 5822.500000 7.151423 5811.0 5819.0 5825.0 5827.0 5831.0
    aflplusplus_afluse_ubsan 65700 2.0 5820.000000 2.828427 5818.0 5819.0 5820.0 5821.0 5822.0
    aflplusplus_afluse_msan 65700 4.0 5812.000000 6.000000 5809.0 5809.0 5809.0 5812.0 5821.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

openthread_ot-ip6-send-fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus_afluse_asan 65700 6.0 3113.666667 140.112336 3040.0 3054.75 3061.5 3066.00 3399.0
    aflplusplus_sanflags_asan 65700 6.0 3146.333333 238.997629 3041.0 3043.50 3053.0 3055.75 3634.0
    aflplusplus_afluse_msan 65700 6.0 3119.500000 174.230594 3042.0 3046.25 3048.5 3055.25 3475.0
    aflplusplus_afluse_ubsan 65700 4.0 3007.000000 54.295488 2926.0 3004.00 3030.0 3033.00 3042.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

proj4_proj_crs_to_crs_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus_afluse_ubsan 65700 8.0 6999.000000 117.065305 6862.0 6919.25 6979.0 7034.75 7202.0
    aflplusplus_afluse_asan 65700 3.0 6551.666667 209.079730 6346.0 6445.50 6545.0 6654.50 6764.0
    aflplusplus_sanflags_asan 65700 4.0 6438.500000 122.077844 6262.0 6412.75 6475.5 6501.25 6541.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

re2_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus_afluse_ubsan 32400 6.0 2874.000000 4.000000 2869.0 2870.75 2874.5 2876.75 2879.0
    aflplusplus_afluse_asan 32400 5.0 2861.800000 15.990622 2834.0 2865.00 2867.0 2868.00 2875.0
    aflplusplus_sanflags_asan 32400 7.0 2865.428571 4.755949 2861.0 2861.00 2864.0 2870.00 2871.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

sqlite3_ossfuzz summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus_afluse_asan 65700 6.0 18498.333333 1024.077862 16994.0 17804.50 18925.5 19209.5 19410.0
    aflplusplus_sanflags_asan 65700 5.0 17862.800000 474.310236 17291.0 17597.00 17927.0 17934.0 18565.0
    aflplusplus_afluse_ubsan 65700 8.0 18008.875000 1477.577403 15654.0 17008.25 17892.5 19494.5 19725.0
    aflplusplus_afluse_msan 65700 4.0 15982.000000 1658.917920 14454.0 14961.00 15607.5 16628.5 18259.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

stb_stbi_read_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus_afluse_asan 65700 6.0 2117.0 29.832868 2079.0 2110.25 2113.0 2115.75 2171.0
    aflplusplus_sanflags_asan 65700 3.0 2099.0 16.643317 2080.0 2093.00 2106.0 2108.50 2111.0
    aflplusplus_afluse_ubsan 65700 6.0 2097.5 14.474115 2083.0 2084.50 2098.0 2110.00 2112.0
    aflplusplus_afluse_msan 65700 5.0 2090.6 13.722245 2083.0 2083.00 2086.0 2086.00 2115.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

vorbis_decode_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus_afluse_msan 65700 9.0 1267.777778 3.073181 1264.0 1265.00 1268.0 1269.0 1272.0
    aflplusplus_sanflags_asan 65700 4.0 1266.500000 1.914854 1265.0 1265.00 1266.0 1267.5 1269.0
    aflplusplus_afluse_asan 65700 8.0 1263.000000 3.505098 1258.0 1261.75 1263.5 1264.5 1268.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

woff2_convert_woff2ttf_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus_afluse_asan 65700 9.0 1164.111111 16.335884 1128.0 1159.0 1172.0 1174.0 1183.0
    aflplusplus_sanflags_asan 65700 6.0 1167.000000 11.471704 1153.0 1160.5 1164.5 1173.0 1185.0
    aflplusplus_afluse_msan 65700 7.0 1155.714286 23.106379 1129.0 1137.0 1157.0 1169.5 1191.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

zlib_zlib_uncompress_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus_sanflags_asan 65700 5.0 462.000000 6.041523 456.0 458.00 460.0 465.00 471.0
    aflplusplus_afluse_ubsan 65700 7.0 458.142857 2.853569 454.0 456.00 459.0 460.00 462.0
    aflplusplus_afluse_asan 65700 6.0 458.166667 3.600926 453.0 456.25 458.0 460.50 463.0
    aflplusplus_afluse_msan 65700 6.0 458.166667 2.639444 456.0 456.25 457.5 458.75 463.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

experiment data

You can download the raw data for this report here.

Check out the documentation on how to create customized reports using this data. Also see some example Colab notebooks for doing custom analysis on the data here.

Experiment Description:

(None,)