FuzzBench: 2023-12-06-fishfuzz report

(experiment incomplete/still running...)

experiment summary

We show two different aggregate (cross-benchmark) rankings of fuzzers. The first is based on the average of per-benchmarks scores, where the score represents the percentage of the highest reached median code-coverage on a given benchmark (higher value is better). The second ranking shows the average rank of fuzzers, after we rank them on each benchmark according to their median reached code-covereges (lower value is better).
By avg. score
average normalized score
fuzzer
aflplusplus 98.66
fishpp_new 95.23
fishpp_new_nocmp 82.91
aflplusplus_nocmp 82.57
By avg. rank
average rank
fuzzer
aflplusplus 1.64
fishpp_new 2.00
fishpp_new_nocmp 2.86
aflplusplus_nocmp 3.09
  • Critical difference diagram
    The diagram visualizes the average rank of fuzzers (second ranking above) while showing the significance of the differences as well. What is considered a "critical difference" (CD) is based on the Friedman/Nemenyi post-hoc test. See more in the documentation.
    Note: If a fuzzer does not support all benchmarks, its ranking as shown in this diagram can be lower than it should be. So please check the list of supported benchmarks for the fuzzer(s) of your interest. The list could be specified in the fuzzer's README.md like this.
  • Median relative code-coverages on each benchmark

    Note: The relative coverage summary table shows the median relative performance of each fuzzer to the experiment maximum. Thus the highest relative performance may not be 100%.
    trial_relative_coverage = trial_coverage / experiment_max_coverage

      fishpp_new aflplusplus fishpp_new_nocmp aflplusplus_nocmp
    FuzzerMedian 97.00 97.50 96.00 95.00
    FuzzerMean 95.67 94.73 83.38 83.00
    bloaty_fuzz_target 99.00 98.00 98.00 98.00
    curl_curl_fuzzer_http 96.00 96.00 nan 93.00
    freetype2_ftfuzzer 97.00 97.00 67.00 70.00
    harfbuzz_hb-shape-fuzzer 98.00 99.00 98.00 99.00
    jsoncpp_jsoncpp_fuzzer 100.00 100.00 99.00 99.00
    lcms_cms_transform_fuzzer 95.00 91.00 81.00 78.00
    libjpeg-turbo_libjpeg_turbo_fuzzer 99.00 99.00 99.00 99.00
    libpcap_fuzz_both 94.00 93.00 1.00 1.00
    libpng_libpng_read_fuzzer 96.00 96.00 96.00 96.00
    libxml2_xml 99.00 99.00 99.00 99.00
    libxslt_xpath nan 98.00 97.00 97.00
    mbedtls_fuzz_dtlsclient 72.00 72.00 71.00 71.00
    openssl_x509 99.00 99.00 99.00 99.00
    openthread_ot-ip6-send-fuzzer 94.00 84.00 78.00 78.00
    proj4_proj_crs_to_crs_fuzzer 94.00 96.00 10.00 10.00
    re2_fuzzer 99.00 99.00 99.00 nan
    sqlite3_ossfuzz 90.00 80.00 85.00 81.00
    stb_stbi_read_fuzzer 95.00 95.00 90.00 90.00
    systemd_fuzz-link-parser 99.00 99.00 94.00 94.00
    vorbis_decode_fuzzer 99.00 99.00 99.00 99.00
    woff2_convert_woff2ttf_fuzzer 98.00 97.00 94.00 95.00
    zlib_zlib_uncompress_fuzzer 97.00 98.00 97.00 97.00
    • Fuzzers are sorted by "FuzzerMean" (average median relative coverage), highest on the left.
    • Green background = highest relative median coverage.
    • Blue gradient background = greater than 95% relative median coverage.

bloaty_fuzz_target summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    fishpp_new 82800 9.0 6385.333333 64.212538 6240.0 6374.00 6404.0 6405.00 6468.0
    fishpp_new_nocmp 82800 8.0 6397.250000 34.820971 6329.0 6385.50 6401.5 6419.75 6435.0
    aflplusplus_nocmp 82800 12.0 6345.916667 96.939023 6057.0 6366.00 6374.0 6387.50 6413.0
    aflplusplus 82800 14.0 6350.571429 51.876650 6252.0 6315.75 6354.0 6371.75 6437.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

curl_curl_fuzzer_http summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    fishpp_new 19800 7.0 10698.571429 50.960302 10628.0 10662.00 10711.0 10728.00 10771.0
    aflplusplus 19800 18.0 10696.333333 47.300044 10605.0 10668.00 10690.5 10727.75 10786.0
    aflplusplus_nocmp 19800 16.0 10405.500000 34.871192 10335.0 10392.75 10411.0 10423.25 10479.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

freetype2_ftfuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    fishpp_new 82800 11.0 11579.818182 266.421778 11089.0 11418.5 11680.0 11807.0 11866.0
    aflplusplus 82800 13.0 11520.076923 303.469510 10801.0 11400.0 11632.0 11694.0 11954.0
    aflplusplus_nocmp 82800 11.0 8293.909091 227.453492 7817.0 8290.0 8383.0 8423.5 8494.0
    fishpp_new_nocmp 82800 7.0 8145.857143 312.148698 7751.0 7998.0 8015.0 8342.0 8575.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

harfbuzz_hb-shape-fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus_nocmp 81900 13.0 10909.384615 166.422424 10476.0 10892.00 10948.0 11014.00 11040.0
    aflplusplus 81900 16.0 10927.062500 69.374797 10769.0 10874.75 10947.0 10966.25 11025.0
    fishpp_new_nocmp 81900 13.0 10911.076923 42.681888 10827.0 10888.00 10921.0 10945.00 10971.0
    fishpp_new 81900 12.0 10722.583333 592.705577 8846.0 10839.00 10890.0 10924.50 10961.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

jsoncpp_jsoncpp_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus 82800 15.0 520.000000 0.000000 520.0 520.00 520.0 520.0 520.0
    fishpp_new 82800 8.0 518.375000 4.206712 508.0 519.75 520.0 520.0 520.0
    aflplusplus_nocmp 82800 8.0 519.000000 0.000000 519.0 519.00 519.0 519.0 519.0
    fishpp_new_nocmp 82800 9.0 515.111111 6.173420 505.0 509.00 519.0 519.0 520.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

lcms_cms_transform_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    fishpp_new 81900 13.0 1897.000000 260.247318 1561.0 1571.00 2043.0 2109.00 2148.0
    aflplusplus 81900 12.0 1863.916667 230.849206 1525.0 1586.25 1963.5 2056.50 2084.0
    fishpp_new_nocmp 81900 15.0 1689.133333 134.141645 1435.0 1554.00 1753.0 1775.00 1860.0
    aflplusplus_nocmp 81900 16.0 1673.937500 111.654508 1500.0 1565.25 1685.0 1776.25 1807.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libjpeg-turbo_libjpeg_turbo_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus 81900 16.0 3083.625000 1.892969 3080.0 3082.0 3084.0 3085.0 3086.0
    fishpp_new_nocmp 81900 15.0 3081.333333 5.752846 3064.0 3081.5 3083.0 3084.5 3086.0
    aflplusplus_nocmp 81900 16.0 3081.562500 3.846535 3072.0 3079.0 3082.5 3084.0 3086.0
    fishpp_new 81900 16.0 3073.750000 12.751471 3054.0 3061.5 3082.5 3084.0 3087.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libpcap_fuzz_both summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    fishpp_new 82800 10.0 2967.400000 103.614885 2801.0 2888.75 2991.0 3031.0 3120.0
    aflplusplus 82800 13.0 2932.692308 118.893078 2713.0 2853.00 2955.0 2990.0 3176.0
    fishpp_new_nocmp 82800 9.0 40.222222 3.833333 33.0 42.00 42.0 42.0 43.0
    aflplusplus_nocmp 82800 8.0 36.375000 4.657943 33.0 33.00 33.0 42.0 42.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libpng_libpng_read_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus 82800 17.0 2008.235294 13.544046 2001.0 2004.00 2005.0 2006.00 2060.0
    fishpp_new 82800 10.0 1995.600000 43.474641 1892.0 1993.75 2001.0 2005.00 2072.0
    aflplusplus_nocmp 82800 6.0 2000.500000 2.073644 1998.0 1999.25 2000.5 2001.00 2004.0
    fishpp_new_nocmp 82800 10.0 1971.900000 46.971504 1898.0 1933.00 2000.0 2001.75 2003.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libxml2_xml summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus 82800 15.0 15740.933333 48.121077 15649.0 15708.00 15753.0 15767.0 15815.0
    aflplusplus_nocmp 82800 8.0 15724.375000 37.618147 15668.0 15688.75 15736.0 15753.0 15768.0
    fishpp_new_nocmp 82800 7.0 15734.571429 33.310945 15693.0 15710.00 15731.0 15757.5 15783.0
    fishpp_new 82800 11.0 15723.545455 38.011482 15662.0 15708.00 15717.0 15741.5 15794.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libxslt_xpath summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus 42300 15.0 11191.933333 105.500485 11015.0 11117.0 11229.0 11269.50 11338.0
    fishpp_new_nocmp 42300 10.0 11047.300000 82.460832 10912.0 10996.5 11039.0 11103.75 11175.0
    aflplusplus_nocmp 42300 17.0 11074.117647 81.904580 10992.0 11016.0 11034.0 11100.00 11247.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

mbedtls_fuzz_dtlsclient summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus 81900 18.0 2840.666667 184.337925 2725.0 2774.75 2795.0 2834.0 3565.0
    fishpp_new 81900 13.0 2769.000000 55.837562 2626.0 2750.00 2773.0 2804.0 2850.0
    fishpp_new_nocmp 81900 16.0 2795.187500 284.052394 2597.0 2716.00 2735.5 2753.0 3850.0
    aflplusplus_nocmp 81900 17.0 2823.470588 241.562134 2707.0 2720.00 2734.0 2774.0 3531.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

openssl_x509 summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    fishpp_new 81900 18.0 5824.333333 7.062744 5806.0 5820.75 5826.5 5828.75 5833.0
    aflplusplus 81900 15.0 5823.733333 3.058166 5818.0 5821.00 5825.0 5825.50 5829.0
    aflplusplus_nocmp 81900 15.0 5821.400000 6.288311 5811.0 5818.50 5823.0 5825.00 5831.0
    fishpp_new_nocmp 81900 12.0 5818.750000 10.323187 5799.0 5812.50 5821.5 5826.00 5831.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

openthread_ot-ip6-send-fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    fishpp_new 9900 1.0 3418.000000 NaN 3418.0 3418.00 3418.0 3418.0 3418.0
    aflplusplus 9900 19.0 3057.842105 78.942499 3011.0 3033.50 3039.0 3044.5 3372.0
    aflplusplus_nocmp 9900 18.0 2836.333333 57.578591 2798.0 2815.25 2820.5 2823.0 3032.0
    fishpp_new_nocmp 9900 4.0 2811.000000 3.915780 2806.0 2809.00 2811.5 2813.5 2815.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

proj4_proj_crs_to_crs_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus 81900 15.0 7153.400000 197.377014 6676.0 7060.0 7172.0 7286.5 7419.0
    fishpp_new 81900 14.0 7073.714286 163.695509 6867.0 6956.0 7035.0 7154.0 7389.0
    fishpp_new_nocmp 81900 15.0 811.600000 4.954075 807.0 807.5 809.0 814.5 822.0
    aflplusplus_nocmp 81900 11.0 801.727273 22.786360 734.0 806.0 807.0 807.5 817.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

re2_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus 82800 15.0 2877.333333 4.546061 2869.0 2875.50 2877.0 2881.00 2884.0
    fishpp_new_nocmp 82800 8.0 2864.625000 33.644942 2782.0 2873.75 2877.0 2878.25 2881.0
    fishpp_new 82800 8.0 2864.625000 32.723681 2784.0 2872.75 2875.5 2878.25 2880.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

sqlite3_ossfuzz summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    fishpp_new 41400 3.0 19221.000000 833.433861 18623.0 18745.00 18867.0 19520.00 20173.0
    fishpp_new_nocmp 41400 3.0 16682.000000 3019.479922 13264.0 15529.50 17795.0 18391.00 18987.0
    aflplusplus_nocmp 41400 18.0 17079.333333 1554.172146 15317.0 15847.00 16933.5 17693.00 20052.0
    aflplusplus 41400 16.0 17266.875000 1455.793066 15442.0 16201.75 16689.0 18672.25 19591.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

stb_stbi_read_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus 82800 13.0 2157.923077 49.614281 2111.0 2114.00 2121.0 2213.00 2219.0
    fishpp_new 82800 14.0 2134.571429 40.912128 2110.0 2113.00 2115.0 2116.00 2211.0
    fishpp_new_nocmp 82800 11.0 2009.545455 1.368476 2007.0 2009.00 2009.0 2010.50 2012.0
    aflplusplus_nocmp 82800 2.0 2007.500000 3.535534 2005.0 2006.25 2007.5 2008.75 2010.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

vorbis_decode_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus 82800 17.0 1267.764706 3.562468 1262.0 1265.00 1268.0 1271.00 1274.0
    fishpp_new_nocmp 82800 15.0 1264.866667 13.542877 1217.0 1266.00 1267.0 1268.50 1276.0
    aflplusplus_nocmp 82800 2.0 1266.500000 2.121320 1265.0 1265.75 1266.5 1267.25 1268.0
    fishpp_new 82800 11.0 1266.090909 4.548726 1261.0 1262.50 1265.0 1267.50 1275.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

woff2_convert_woff2ttf_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    fishpp_new 81900 13.0 1156.384615 40.699178 1060.0 1158.0 1175.0 1181.00 1191.0
    aflplusplus 81900 15.0 1172.866667 19.171656 1131.0 1164.5 1171.0 1191.00 1195.0
    aflplusplus_nocmp 81900 18.0 1136.388889 5.689401 1128.0 1132.0 1136.0 1141.75 1147.0
    fishpp_new_nocmp 81900 13.0 1125.769231 18.934773 1065.0 1127.0 1130.0 1136.00 1137.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

zlib_zlib_uncompress_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus 81900 16.0 460.375000 3.667424 456.0 457.0 460.0 462.25 469.0
    aflplusplus_nocmp 81900 16.0 459.687500 3.156343 456.0 457.0 459.5 460.50 469.0
    fishpp_new_nocmp 81900 16.0 459.687500 3.807339 455.0 457.0 459.5 462.00 468.0
    fishpp_new 81900 13.0 460.461538 4.701336 455.0 457.0 459.0 463.00 469.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

experiment data

You can download the raw data for this report here.

Check out the documentation on how to create customized reports using this data. Also see some example Colab notebooks for doing custom analysis on the data here.

Experiment Description:

(None,)