FuzzBench: 2023-09-28-libafl report

(experiment incomplete/still running...)

experiment summary

We show two different aggregate (cross-benchmark) rankings of fuzzers. The first is based on the average of per-benchmarks scores, where the score represents the percentage of the highest reached median code-coverage on a given benchmark (higher value is better). The second ranking shows the average rank of fuzzers, after we rank them on each benchmark according to their median reached code-covereges (lower value is better).
By avg. score
average normalized score
fuzzer
libafl_27042023 99.43
libafl 99.01
aflrustrust 98.90
By avg. rank
average rank
fuzzer
libafl 1.61
aflrustrust 2.09
libafl_27042023 2.17
  • Critical difference diagram
    The diagram visualizes the average rank of fuzzers (second ranking above) while showing the significance of the differences as well. What is considered a "critical difference" (CD) is based on the Friedman/Nemenyi post-hoc test. See more in the documentation.
    Note: If a fuzzer does not support all benchmarks, its ranking as shown in this diagram can be lower than it should be. So please check the list of supported benchmarks for the fuzzer(s) of your interest. The list could be specified in the fuzzer's README.md like this.
  • Median relative code-coverages on each benchmark

    Note: The relative coverage summary table shows the median relative performance of each fuzzer to the experiment maximum. Thus the highest relative performance may not be 100%.
    trial_relative_coverage = trial_coverage / experiment_max_coverage

      libafl_27042023 libafl aflrustrust
    FuzzerMedian 98.00 98.00 98.00
    FuzzerMean 95.61 95.17 95.13
    bloaty_fuzz_target 98.00 98.00 92.00
    curl_curl_fuzzer_http 98.00 98.00 98.00
    freetype2_ftfuzzer 86.00 89.00 85.00
    harfbuzz_hb-shape-fuzzer 99.00 99.00 99.00
    jsoncpp_jsoncpp_fuzzer 99.00 99.00 100.00
    lcms_cms_transform_fuzzer 90.00 91.00 89.00
    libjpeg-turbo_libjpeg_turbo_fuzzer 99.00 85.00 99.00
    libpcap_fuzz_both 89.00 89.00 89.00
    libpng_libpng_read_fuzzer 98.00 99.00 98.00
    libxml2_xml 99.00 98.00 99.00
    libxslt_xpath 97.00 98.00 97.00
    mbedtls_fuzz_dtlsclient 87.00 83.00 84.00
    openh264_decoder_fuzzer 99.00 99.00 99.00
    openssl_x509 99.00 99.00 99.00
    openthread_ot-ip6-send-fuzzer 85.00 85.00 85.00
    proj4_proj_crs_to_crs_fuzzer 94.00 95.00 95.00
    re2_fuzzer 99.00 99.00 99.00
    sqlite3_ossfuzz 97.00 98.00 97.00
    stb_stbi_read_fuzzer 96.00 96.00 93.00
    systemd_fuzz-link-parser 99.00 99.00 98.00
    vorbis_decode_fuzzer 98.00 98.00 99.00
    woff2_convert_woff2ttf_fuzzer 98.00 98.00 98.00
    zlib_zlib_uncompress_fuzzer 96.00 97.00 97.00
    • Fuzzers are sorted by "FuzzerMean" (average median relative coverage), highest on the left.
    • Green background = highest relative median coverage.
    • Blue gradient background = greater than 95% relative median coverage.

bloaty_fuzz_target summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libafl_27042023 82800 20.0 6371.200000 45.055054 6289.0 6340.25 6376.5 6402.00 6443.0
    libafl 82800 19.0 6366.631579 57.494164 6218.0 6331.50 6362.0 6405.00 6463.0
    aflrustrust 82800 20.0 6026.400000 78.122306 5874.0 5987.25 6006.0 6062.25 6180.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

curl_curl_fuzzer_http summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflrustrust 82800 20.0 10749.050000 58.534268 10674.0 10698.50 10741.0 10777.5 10899.0
    libafl 82800 18.0 10727.333333 47.288850 10654.0 10688.25 10732.5 10743.5 10829.0
    libafl_27042023 82800 19.0 10684.578947 50.591299 10613.0 10646.00 10690.0 10715.5 10793.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

freetype2_ftfuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libafl 82800 20.0 11451.650000 648.833469 10318.0 11019.25 11363.0 11953.75 12703.0
    libafl_27042023 82800 20.0 10969.700000 518.498652 9978.0 10664.50 10993.5 11290.00 12298.0
    aflrustrust 82800 19.0 10834.210526 563.171336 9792.0 10514.50 10814.0 11041.50 12210.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

harfbuzz_hb-shape-fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libafl 82800 20.0 11081.900000 33.790220 11028.0 11056.75 11083.0 11101.75 11146.0
    aflrustrust 82800 19.0 11058.684211 34.722795 11002.0 11031.00 11058.0 11077.50 11125.0
    libafl_27042023 82800 20.0 11045.800000 27.804960 11007.0 11024.00 11042.5 11061.00 11092.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

jsoncpp_jsoncpp_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflrustrust 82800 20.0 518.600000 0.598243 517.0 518.0 519.0 519.0 519.0
    libafl 82800 18.0 517.111111 0.832352 516.0 517.0 517.0 517.0 519.0
    libafl_27042023 82800 20.0 517.050000 0.686333 516.0 517.0 517.0 517.0 519.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

lcms_cms_transform_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libafl 82800 20.0 2068.85 73.730145 1958.0 2020.00 2074.0 2118.00 2270.0
    libafl_27042023 82800 20.0 2043.75 113.528051 1645.0 2013.75 2052.5 2104.75 2206.0
    aflrustrust 82800 20.0 2033.40 133.260094 1586.0 1999.00 2036.0 2132.75 2204.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libjpeg-turbo_libjpeg_turbo_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflrustrust 82800 20.0 3075.550000 15.517308 3034.0 3079.5 3082.0 3083.0 3084.0
    libafl_27042023 82800 19.0 3078.210526 3.629799 3072.0 3074.5 3079.0 3081.0 3083.0
    libafl 82800 20.0 2667.900000 91.720342 2537.0 2610.5 2642.5 2727.0 2849.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libpcap_fuzz_both summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libafl 82800 20.0 2793.850000 136.223607 2632.0 2669.5 2745.5 2902.25 3051.0
    libafl_27042023 82800 19.0 2698.894737 126.633283 2450.0 2618.0 2725.0 2802.50 2868.0
    aflrustrust 82800 20.0 2719.900000 106.619787 2564.0 2640.0 2720.0 2757.75 2951.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libpng_libpng_read_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libafl 82800 20.0 1990.35 11.041906 1973.0 1979.75 1996.5 1999.25 2003.0
    aflrustrust 82800 20.0 1987.70 9.695903 1976.0 1981.00 1983.0 1995.50 2004.0
    libafl_27042023 82800 20.0 1985.25 12.069426 1970.0 1975.00 1979.5 1998.50 2001.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libxml2_xml summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflrustrust 82800 19.0 15593.894737 25.607930 15555.0 15585.5 15598.0 15604.00 15649.0
    libafl_27042023 82800 20.0 15244.550000 1514.482627 8811.0 15565.0 15576.5 15598.25 15632.0
    libafl 82800 19.0 15331.263158 221.420626 14890.0 15188.5 15356.0 15482.00 15668.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libxslt_xpath summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libafl 82800 19.0 10955.052632 64.621525 10839.0 10921.5 10939.0 10985.50 11128.0
    libafl_27042023 82800 19.0 10829.052632 53.489847 10712.0 10798.0 10838.0 10869.50 10908.0
    aflrustrust 82800 20.0 10841.800000 96.329260 10691.0 10783.5 10814.5 10932.25 10991.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

mbedtls_fuzz_dtlsclient summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libafl_27042023 82800 20.0 3161.25 325.753681 2688.0 2748.75 3263.0 3393.75 3553.0
    aflrustrust 82800 20.0 3087.65 339.094037 2703.0 2720.00 3143.0 3370.50 3718.0
    libafl 82800 20.0 3086.75 343.855214 2712.0 2750.75 3109.5 3370.00 3685.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

openh264_decoder_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflrustrust 82800 20.0 9539.450000 7.639751 9519.0 9535.75 9539.0 9544.25 9554.0
    libafl 82800 18.0 9537.666667 6.435197 9523.0 9533.25 9538.5 9541.25 9548.0
    libafl_27042023 82800 20.0 9531.100000 18.940072 9473.0 9527.75 9535.5 9540.50 9555.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

openssl_x509 summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libafl 82800 19.0 5826.052632 4.287886 5821.0 5822.0 5825.0 5831.0 5831.0
    libafl_27042023 82800 19.0 5824.789474 5.370485 5809.0 5822.0 5825.0 5829.0 5831.0
    aflrustrust 82800 20.0 5823.750000 6.843168 5809.0 5821.0 5824.5 5828.5 5834.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

openthread_ot-ip6-send-fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflrustrust 82800 19.0 3417.263158 325.088679 3018.0 3051.0 3533.0 3561.50 4047.0
    libafl_27042023 82800 20.0 3451.000000 304.559216 3042.0 3058.5 3532.5 3578.25 3993.0
    libafl 82800 20.0 3405.800000 367.643363 3030.0 3050.0 3519.5 3570.75 4139.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

proj4_proj_crs_to_crs_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libafl 82800 20.0 7302.100000 103.638999 7066.0 7252.5 7293.5 7368.75 7508.0
    aflrustrust 82800 19.0 7271.000000 196.652259 6951.0 7134.5 7265.0 7404.00 7639.0
    libafl_27042023 82800 19.0 7187.947368 151.312948 6790.0 7157.0 7234.0 7264.00 7389.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

re2_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libafl 82800 18.0 2855.666667 6.624376 2842.0 2852.0 2854.0 2861.75 2867.0
    aflrustrust 82800 20.0 2854.350000 7.548266 2841.0 2848.0 2853.0 2858.25 2869.0
    libafl_27042023 82800 20.0 2849.250000 6.827768 2838.0 2845.0 2848.0 2852.25 2863.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

sqlite3_ossfuzz summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libafl 82800 20.0 18483.75 136.161234 18205.0 18398.75 18473.5 18558.00 18735.0
    aflrustrust 82800 20.0 18324.40 132.625472 18136.0 18211.50 18316.0 18371.00 18677.0
    libafl_27042023 82800 20.0 18214.55 153.367970 17944.0 18121.00 18185.5 18297.75 18548.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

stb_stbi_read_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libafl_27042023 82800 20.0 2178.350000 46.130564 2101.0 2142.75 2190.0 2195.00 2263.0
    libafl 82800 19.0 2167.578947 48.597343 2070.0 2137.00 2187.0 2193.00 2266.0
    aflrustrust 82800 20.0 2101.850000 49.296391 2018.0 2084.50 2110.0 2117.25 2190.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

vorbis_decode_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflrustrust 82800 20.0 1263.650000 2.455391 1258.0 1263.00 1264.0 1265.0 1271.0
    libafl 82800 20.0 1251.700000 4.341962 1240.0 1248.75 1252.5 1254.0 1259.0
    libafl_27042023 82800 19.0 1251.263158 3.015554 1247.0 1249.50 1251.0 1253.0 1259.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

woff2_convert_woff2ttf_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libafl 82800 19.0 1175.736842 12.622036 1151.0 1166.50 1179.0 1185.50 1196.0
    libafl_27042023 82800 19.0 1151.842105 88.809574 791.0 1153.00 1177.0 1183.00 1194.0
    aflrustrust 82800 20.0 1168.200000 16.227332 1140.0 1152.25 1172.5 1180.25 1188.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

zlib_zlib_uncompress_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libafl 82800 20.0 450.400000 5.432844 440.0 447.50 449.5 454.5 460.0
    aflrustrust 82800 19.0 447.105263 5.184322 440.0 442.50 448.0 451.5 455.0
    libafl_27042023 82800 20.0 444.550000 4.477723 435.0 442.25 445.0 448.0 451.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

experiment data

You can download the raw data for this report here.

Check out the documentation on how to create customized reports using this data. Also see some example Colab notebooks for doing custom analysis on the data here.

Experiment Description:

(None,)