FuzzBench: 2025-04-01-libafl-scheduling1 report

(experiment incomplete/still running...)

experiment summary

We show two different aggregate (cross-benchmark) rankings of fuzzers. The first is based on the average of per-benchmarks scores, where the score represents the percentage of the highest reached median code-coverage on a given benchmark (higher value is better). The second ranking shows the average rank of fuzzers, after we rank them on each benchmark according to their median reached code-covereges (lower value is better).
By avg. score
average normalized score
fuzzer
libafl_fast 97.66
libafl_covaccount 93.79
libafl_naive 93.35
libafl_random 91.73
libafl_weighted 90.88
libafl_explore 90.80
By avg. rank
average rank
fuzzer
libafl_fast 2.71
libafl_explore 2.81
libafl_naive 3.06
libafl_random 3.19
libafl_weighted 3.35
libafl_covaccount 3.77
  • Critical difference diagram
    The diagram visualizes the average rank of fuzzers (second ranking above) while showing the significance of the differences as well. What is considered a "critical difference" (CD) is based on the Friedman/Nemenyi post-hoc test. See more in the documentation.
    Note: If a fuzzer does not support all benchmarks, its ranking as shown in this diagram can be lower than it should be. So please check the list of supported benchmarks for the fuzzer(s) of your interest. The list could be specified in the fuzzer's README.md like this.
  • Median relative code-coverages on each benchmark

    Note: The relative coverage summary table shows the median relative performance of each fuzzer to the experiment maximum. Thus the highest relative performance may not be 100%.
    trial_relative_coverage = trial_coverage / experiment_max_coverage

      libafl_explore libafl_fast libafl_weighted libafl_random libafl_covaccount libafl_naive
    FuzzerMedian 95.00 95.00 93.50 92.00 90.00 92.00
    FuzzerMean 87.62 87.45 86.77 84.47 83.61 83.45
    bloaty_fuzz_target 100.00 100.00 100.00 100.00 100.00 100.00
    curl_curl_fuzzer_http 62.00 62.00 61.00 62.00 79.00 61.00
    freetype2_ftfuzzer nan 89.00 88.00 83.00 85.00 88.00
    harfbuzz_hb-shape-fuzzer 99.00 98.00 99.00 99.00 99.00 99.00
    jsoncpp_jsoncpp_fuzzer 99.00 99.00 99.00 99.00 99.00 99.00
    lcms_cms_transform_fuzzer 7.00 7.00 nan 7.00 7.00 7.00
    libaom_av1_dec_fuzzer 98.00 95.00 94.00 81.00 64.00 94.00
    libjpeg-turbo_libjpeg_turbo_fuzzer 24.00 24.00 25.00 25.00 25.00 27.00
    libpcap_fuzz_both 91.00 89.00 91.00 90.00 94.00 93.00
    libpng_libpng_read_fuzzer 100.00 100.00 100.00 100.00 100.00 100.00
    libvpx_vpx_dec_fuzzer_vp8 98.00 98.00 95.00 93.00 96.00 96.00
    libvpx_vpx_dec_fuzzer_vp9 99.00 98.00 98.00 95.00 93.00 99.00
    libxml2_xml 89.00 94.00 91.00 70.00 63.00 69.00
    libxslt_xpath 98.00 98.00 98.00 nan 98.00 99.00
    mbedtls_fuzz_dtlsclient 91.00 91.00 91.00 94.00 90.00 91.00
    mruby_mruby_fuzzer 97.00 91.00 78.00 85.00 88.00 92.00
    openssl_x509 94.00 97.00 73.00 96.00 73.00 80.00
    openthread_ot-ip6-send-fuzzer 95.00 95.00 95.00 95.00 92.00 92.00
    php_php-fuzz-execute 94.00 96.00 91.00 82.00 81.00 79.00
    php_php-fuzz-exif 98.00 97.00 97.00 97.00 97.00 97.00
    php_php-fuzz-parser 95.00 97.00 93.00 91.00 87.00 89.00
    php_php-fuzz-unserialize 91.00 90.00 95.00 95.00 93.00 90.00
    php_php-fuzz-unserializehash 94.00 96.00 94.00 98.00 93.00 98.00
    proj4_proj_crs_to_crs_fuzzer 88.00 87.00 87.00 80.00 83.00 81.00
    re2_fuzzer 99.00 99.00 99.00 99.00 99.00 99.00
    sqlite3_ossfuzz 75.00 87.00 75.00 75.00 74.00 75.00
    systemd_fuzz-link-parser 100.00 100.00 100.00 100.00 100.00 100.00
    vorbis_decode_fuzzer 82.00 91.00 57.00 89.00 88.00 50.00
    wireshark_fuzzshark nan 69.00 54.00 68.00 68.00 55.00
    woff2_convert_woff2ttf_fuzzer 89.00 80.00 90.00 90.00 89.00 91.00
    zlib_zlib_uncompress_fuzzer 95.00 97.00 95.00 96.00 95.00 97.00
    • Fuzzers are sorted by "FuzzerMean" (average median relative coverage), highest on the left.
    • Green background = highest relative median coverage.
    • Blue gradient background = greater than 95% relative median coverage.

bloaty_fuzz_target summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libafl_covaccount 60300 6.0 980.833333 0.408248 980.0 981.00 981.0 981.0 981.0
    libafl_explore 60300 4.0 980.750000 0.500000 980.0 980.75 981.0 981.0 981.0
    libafl_fast 60300 3.0 981.000000 0.000000 981.0 981.00 981.0 981.0 981.0
    libafl_naive 60300 3.0 980.666667 0.577350 980.0 980.50 981.0 981.0 981.0
    libafl_random 60300 2.0 981.000000 0.000000 981.0 981.00 981.0 981.0 981.0
    libafl_weighted 60300 4.0 981.000000 0.000000 981.0 981.00 981.0 981.0 981.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

curl_curl_fuzzer_http summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libafl_covaccount 54900 4.0 6649.25 1667.683498 5201.0 5207.0 6649.0 8091.25 8098.0
    libafl_explore 54900 3.0 6088.00 1458.737811 5214.0 5246.0 5278.0 6525.00 7772.0
    libafl_random 54900 1.0 5229.00 NaN 5229.0 5229.0 5229.0 5229.00 5229.0
    libafl_fast 54900 3.0 5219.00 4.582576 5215.0 5216.5 5218.0 5221.00 5224.0
    libafl_naive 54900 3.0 5210.00 1.000000 5209.0 5209.5 5210.0 5210.50 5211.0
    libafl_weighted 54900 6.0 5688.00 1157.003371 5189.0 5202.0 5209.5 5256.00 8049.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

freetype2_ftfuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libafl_fast 63900 7.0 5465.142857 255.466520 5273.0 5310.50 5372.0 5492.50 6005.0
    libafl_weighted 63900 7.0 5169.857143 554.766446 4023.0 5127.50 5302.0 5414.00 5781.0
    libafl_naive 63900 2.0 5290.000000 270.114790 5099.0 5194.50 5290.0 5385.50 5481.0
    libafl_covaccount 63900 2.0 5105.500000 480.125504 4766.0 4935.75 5105.5 5275.25 5445.0
    libafl_random 63900 2.0 5022.000000 18.384776 5009.0 5015.50 5022.0 5028.50 5035.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

harfbuzz_hb-shape-fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libafl_weighted 62100 5.0 4504.600000 7.569676 4497.0 4500.0 4501.0 4510.0 4515.0
    libafl_random 62100 3.0 4495.333333 10.408330 4487.0 4489.5 4492.0 4499.5 4507.0
    libafl_explore 62100 5.0 4494.800000 12.008330 4485.0 4486.0 4489.0 4501.0 4513.0
    libafl_covaccount 62100 3.0 4478.666667 23.459184 4453.0 4468.5 4484.0 4491.5 4499.0
    libafl_naive 62100 7.0 4474.571429 11.043205 4458.0 4468.5 4476.0 4481.0 4489.0
    libafl_fast 62100 2.0 4469.000000 9.899495 4462.0 4465.5 4469.0 4472.5 4476.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

jsoncpp_jsoncpp_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libafl_random 59400 2.0 519.500000 0.707107 519.0 519.25 519.5 519.75 520.0
    libafl_covaccount 59400 3.0 519.333333 0.577350 519.0 519.00 519.0 519.50 520.0
    libafl_fast 59400 3.0 518.666667 0.577350 518.0 518.50 519.0 519.00 519.0
    libafl_explore 59400 6.0 518.500000 1.048809 517.0 518.00 518.5 519.00 520.0
    libafl_naive 59400 4.0 518.000000 2.160247 515.0 517.25 518.5 519.25 520.0
    libafl_weighted 59400 4.0 518.250000 0.957427 517.0 517.75 518.5 519.00 519.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

lcms_cms_transform_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libafl_covaccount 54900 3.0 36.0 0.0 36.0 36.0 36.0 36.0 36.0
    libafl_explore 54900 5.0 36.0 0.0 36.0 36.0 36.0 36.0 36.0
    libafl_fast 54900 2.0 36.0 0.0 36.0 36.0 36.0 36.0 36.0
    libafl_naive 54900 3.0 36.0 0.0 36.0 36.0 36.0 36.0 36.0
    libafl_random 54900 5.0 36.0 0.0 36.0 36.0 36.0 36.0 36.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libaom_av1_dec_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libafl_explore 63000 6.0 10363.833333 299.235972 9831.0 10282.00 10446.5 10587.75 10597.0
    libafl_fast 63000 5.0 10183.400000 151.987828 10024.0 10096.00 10115.0 10291.00 10391.0
    libafl_weighted 63000 5.0 10106.000000 127.064944 10014.0 10038.00 10063.0 10087.00 10328.0
    libafl_naive 63000 2.0 10035.000000 155.563492 9925.0 9980.00 10035.0 10090.00 10145.0
    libafl_random 63000 2.0 8674.500000 2352.544261 7011.0 7842.75 8674.5 9506.25 10338.0
    libafl_covaccount 63000 3.0 7369.333333 1643.031446 6063.0 6447.00 6831.0 8022.50 9214.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libjpeg-turbo_libjpeg_turbo_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libafl_naive 62100 3.0 537.000000 24.248711 509.0 530.00 551.0 551.00 551.0
    libafl_weighted 62100 1.0 514.000000 NaN 514.0 514.00 514.0 514.00 514.0
    libafl_covaccount 62100 5.0 857.000000 663.704754 509.0 509.00 513.0 721.00 2033.0
    libafl_random 62100 2.0 510.000000 0.000000 510.0 510.00 510.0 510.00 510.0
    libafl_fast 62100 3.0 510.333333 4.163332 507.0 508.00 509.0 512.00 515.0
    libafl_explore 62100 2.0 507.500000 2.121320 506.0 506.75 507.5 508.25 509.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libpcap_fuzz_both summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libafl_covaccount 58500 3.0 2734.666667 42.122836 2689.0 2716.00 2743.0 2757.50 2772.0
    libafl_naive 58500 5.0 2753.400000 63.172779 2682.0 2716.00 2729.0 2820.00 2820.0
    libafl_explore 58500 2.0 2672.000000 186.676190 2540.0 2606.00 2672.0 2738.00 2804.0
    libafl_weighted 58500 3.0 2687.666667 71.626345 2634.0 2647.00 2660.0 2714.50 2769.0
    libafl_random 58500 1.0 2635.000000 NaN 2635.0 2635.00 2635.0 2635.00 2635.0
    libafl_fast 58500 2.0 2592.500000 168.998521 2473.0 2532.75 2592.5 2652.25 2712.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libpng_libpng_read_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libafl_covaccount 54900 2.0 7.0 0.0 7.0 7.0 7.0 7.0 7.0
    libafl_explore 54900 5.0 7.0 0.0 7.0 7.0 7.0 7.0 7.0
    libafl_fast 54900 2.0 7.0 0.0 7.0 7.0 7.0 7.0 7.0
    libafl_naive 54900 8.0 7.0 0.0 7.0 7.0 7.0 7.0 7.0
    libafl_random 54900 2.0 7.0 0.0 7.0 7.0 7.0 7.0 7.0
    libafl_weighted 54900 4.0 7.0 0.0 7.0 7.0 7.0 7.0 7.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libvpx_vpx_dec_fuzzer_vp8 summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libafl_explore 55800 4.0 1306.250000 27.956812 1270.0 1291.75 1311.5 1326.0 1332.0
    libafl_fast 55800 5.0 1274.200000 59.922450 1181.0 1247.00 1306.0 1318.0 1319.0
    libafl_naive 55800 4.0 1268.500000 60.830365 1182.0 1249.50 1286.5 1305.5 1319.0
    libafl_covaccount 55800 5.0 1234.400000 125.663439 1013.0 1268.00 1282.0 1283.0 1326.0
    libafl_weighted 55800 3.0 1276.333333 20.033306 1261.0 1265.00 1269.0 1284.0 1299.0
    libafl_random 55800 3.0 1153.666667 197.221534 928.0 1084.00 1240.0 1266.5 1293.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libvpx_vpx_dec_fuzzer_vp9 summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libafl_explore 58500 4.0 3578.0 18.348479 3551.0 3575.75 3584.5 3586.75 3592.0
    libafl_naive 58500 4.0 3556.0 23.790755 3521.0 3552.50 3564.5 3568.00 3574.0
    libafl_fast 58500 5.0 3553.0 45.972818 3481.0 3543.00 3556.0 3588.00 3597.0
    libafl_weighted 58500 4.0 3550.0 15.641824 3532.0 3539.50 3551.5 3562.00 3565.0
    libafl_random 58500 2.0 3447.0 69.296465 3398.0 3422.50 3447.0 3471.50 3496.0
    libafl_covaccount 58500 4.0 3384.0 78.562077 3297.0 3343.50 3377.0 3417.50 3485.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libxml2_xml summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libafl_fast 57600 4.0 11294.75 272.459630 10934.0 11159.00 11367.5 11503.25 11510.0
    libafl_weighted 57600 4.0 10963.00 278.612036 10578.0 10853.25 11035.0 11144.75 11204.0
    libafl_explore 57600 2.0 10776.50 420.728535 10479.0 10627.75 10776.5 10925.25 11074.0
    libafl_random 57600 5.0 8376.40 295.843371 7937.0 8216.00 8491.0 8604.00 8634.0
    libafl_naive 57600 2.0 8363.00 306.884343 8146.0 8254.50 8363.0 8471.50 8580.0
    libafl_covaccount 57600 5.0 7791.80 447.479832 7482.0 7492.00 7591.0 7846.00 8548.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libxslt_xpath summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libafl_naive 57600 2.0 5659.500000 43.133514 5629.0 5644.25 5659.5 5674.75 5690.0
    libafl_fast 57600 5.0 5635.400000 11.970798 5616.0 5637.00 5637.0 5638.00 5649.0
    libafl_explore 57600 3.0 5635.333333 3.511885 5632.0 5633.50 5635.0 5637.00 5639.0
    libafl_covaccount 57600 6.0 5623.500000 10.014989 5613.0 5615.25 5622.0 5632.50 5635.0
    libafl_weighted 57600 7.0 5628.428571 22.059444 5609.0 5618.00 5621.0 5628.50 5676.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

mbedtls_fuzz_dtlsclient summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libafl_random 54000 2.0 1616.500000 89.802561 1553.0 1584.75 1616.5 1648.25 1680.0
    libafl_explore 54000 2.0 1562.500000 2.121320 1561.0 1561.75 1562.5 1563.25 1564.0
    libafl_naive 54000 1.0 1558.000000 NaN 1558.0 1558.00 1558.0 1558.00 1558.0
    libafl_fast 54000 6.0 1557.333333 3.829708 1553.0 1554.00 1557.5 1560.25 1562.0
    libafl_weighted 54000 5.0 1585.000000 66.562001 1553.0 1553.00 1557.0 1558.00 1704.0
    libafl_covaccount 54000 3.0 1557.000000 6.082763 1553.0 1553.50 1554.0 1559.00 1564.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

mruby_mruby_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libafl_explore 52200 3.0 7110.000000 173.674984 6911.0 7049.5 7188.0 7209.5 7231.0
    libafl_naive 52200 3.0 6783.666667 9.291573 6773.0 6780.5 6788.0 6789.0 6790.0
    libafl_fast 52200 2.0 6748.000000 50.911688 6712.0 6730.0 6748.0 6766.0 6784.0
    libafl_covaccount 52200 4.0 6501.000000 170.190090 6268.0 6463.0 6529.5 6567.5 6677.0
    libafl_random 52200 3.0 6330.000000 186.502011 6196.0 6223.5 6251.0 6397.0 6543.0
    libafl_weighted 52200 5.0 5863.200000 126.410838 5784.0 5788.0 5793.0 5871.0 6080.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

openssl_x509 summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libafl_fast 57600 3.0 2388.333333 59.500700 2322.0 2364.0 2406.0 2421.50 2437.0
    libafl_random 57600 7.0 2216.142857 282.557032 1805.0 2046.5 2364.0 2396.50 2458.0
    libafl_explore 57600 3.0 2186.666667 329.613916 1812.0 2064.0 2316.0 2374.00 2432.0
    libafl_naive 57600 4.0 1946.250000 265.982926 1640.0 1767.5 1973.0 2151.75 2199.0
    libafl_covaccount 57600 3.0 1985.000000 293.586444 1814.0 1815.5 1817.0 2070.50 2324.0
    libafl_weighted 57600 3.0 1915.666667 215.428256 1769.0 1792.0 1815.0 1989.00 2163.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

openthread_ot-ip6-send-fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libafl_explore 58500 6.0 2934.833333 58.060026 2907.0 2908.00 2911.5 2917.25 3053.0
    libafl_random 58500 6.0 2905.166667 77.700493 2818.0 2849.25 2911.5 2924.25 3033.0
    libafl_weighted 58500 6.0 2892.500000 36.396428 2819.0 2900.00 2907.0 2909.50 2914.0
    libafl_fast 58500 3.0 2879.666667 42.158431 2831.0 2867.00 2903.0 2904.00 2905.0
    libafl_naive 58500 3.0 2849.666667 48.013887 2819.0 2822.00 2825.0 2865.00 2905.0
    libafl_covaccount 58500 3.0 2847.000000 45.902070 2820.0 2820.50 2821.0 2860.50 2900.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

php_php-fuzz-execute summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libafl_fast 57600 4.0 18486.500000 748.335264 17494.0 18115.75 18691.0 19061.75 19070.0
    libafl_explore 57600 1.0 18368.000000 NaN 18368.0 18368.00 18368.0 18368.00 18368.0
    libafl_weighted 57600 5.0 17514.000000 936.495328 16013.0 17364.00 17805.0 17860.00 18528.0
    libafl_random 57600 6.0 15931.333333 1056.016035 14500.0 15185.00 16089.0 16732.00 17080.0
    libafl_covaccount 57600 3.0 15687.333333 887.886442 14729.0 15290.00 15851.0 16166.50 16482.0
    libafl_naive 57600 5.0 15648.600000 586.082161 14978.0 15497.00 15534.0 15643.00 16591.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

php_php-fuzz-exif summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libafl_explore 45900 6.0 3992.000000 37.619144 3956.0 3958.00 3989.0 4026.00 4032.0
    libafl_weighted 45900 5.0 3966.400000 27.709204 3945.0 3956.00 3957.0 3959.00 4015.0
    libafl_naive 45900 4.0 3949.000000 17.146428 3931.0 3937.75 3947.5 3958.75 3970.0
    libafl_covaccount 45900 5.0 3953.400000 22.777182 3941.0 3942.00 3944.0 3946.00 3994.0
    libafl_fast 45900 3.0 3944.666667 4.618802 3942.0 3942.00 3942.0 3946.00 3950.0
    libafl_random 45900 5.0 3940.000000 5.291503 3933.0 3937.00 3941.0 3942.00 3947.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

php_php-fuzz-parser summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libafl_fast 57600 3.0 12642.666667 398.684504 12220.0 12458.00 12696.0 12854.00 13012.0
    libafl_explore 57600 3.0 12495.666667 185.974013 12377.0 12388.50 12400.0 12555.00 12710.0
    libafl_weighted 57600 3.0 12214.333333 68.850079 12139.0 12184.50 12230.0 12252.00 12274.0
    libafl_random 57600 4.0 11968.000000 309.499596 11677.0 11836.75 11894.5 12025.75 12406.0
    libafl_naive 57600 2.0 11723.500000 6.363961 11719.0 11721.25 11723.5 11725.75 11728.0
    libafl_covaccount 57600 6.0 11402.833333 103.817950 11204.0 11393.75 11436.0 11469.25 11481.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

php_php-fuzz-unserialize summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libafl_weighted 66600 3.0 3840.333333 41.186567 3793.0 3826.50 3860.0 3864.00 3868.0
    libafl_random 66600 5.0 3869.400000 77.102529 3810.0 3836.00 3840.0 3857.00 4004.0
    libafl_covaccount 66600 4.0 3752.250000 111.896902 3645.0 3661.50 3752.0 3842.75 3860.0
    libafl_explore 66600 3.0 3719.333333 100.530261 3653.0 3661.50 3670.0 3752.50 3835.0
    libafl_naive 66600 3.0 3661.333333 3.055050 3658.0 3660.00 3662.0 3663.00 3664.0
    libafl_fast 66600 2.0 3655.500000 2.121320 3654.0 3654.75 3655.5 3656.25 3657.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

php_php-fuzz-unserializehash summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libafl_random 57600 6.0 3812.0 73.042453 3675.0 3801.00 3836.5 3851.75 3878.0
    libafl_naive 57600 3.0 3770.0 109.412065 3644.0 3734.50 3825.0 3833.00 3841.0
    libafl_fast 57600 4.0 3738.5 118.193344 3617.0 3647.75 3743.0 3833.75 3851.0
    libafl_weighted 57600 2.0 3670.5 3.535534 3668.0 3669.25 3670.5 3671.75 3673.0
    libafl_explore 57600 2.0 3665.5 6.363961 3661.0 3663.25 3665.5 3667.75 3670.0
    libafl_covaccount 57600 1.0 3650.0 NaN 3650.0 3650.00 3650.0 3650.00 3650.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

proj4_proj_crs_to_crs_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libafl_explore 52200 5.0 5117.000000 354.737931 4564.0 5124.0 5140.0 5205.0 5552.0
    libafl_weighted 52200 3.0 4942.000000 319.868723 4576.0 4829.0 5082.0 5125.0 5168.0
    libafl_fast 52200 3.0 5192.666667 467.754565 4802.0 4933.5 5065.0 5388.0 5711.0
    libafl_covaccount 52200 3.0 5024.333333 570.365088 4540.0 4710.0 4880.0 5266.5 5653.0
    libafl_naive 52200 3.0 4716.666667 72.279550 4641.0 4682.5 4724.0 4754.5 4785.0
    libafl_random 52200 4.0 4769.000000 321.864361 4473.0 4546.5 4705.5 4928.0 5192.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

re2_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libafl_naive 52200 2.0 2854.000000 7.071068 2849.0 2851.5 2854.0 2856.5 2859.0
    libafl_weighted 52200 3.0 2851.666667 9.712535 2841.0 2847.5 2854.0 2857.0 2860.0
    libafl_explore 52200 5.0 2851.800000 3.492850 2848.0 2850.0 2850.0 2855.0 2856.0
    libafl_random 52200 1.0 2850.000000 NaN 2850.0 2850.0 2850.0 2850.0 2850.0
    libafl_covaccount 52200 2.0 2842.000000 1.414214 2841.0 2841.5 2842.0 2842.5 2843.0
    libafl_fast 52200 3.0 2843.666667 7.234178 2839.0 2839.5 2840.0 2846.0 2852.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

sqlite3_ossfuzz summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libafl_fast 58500 2.0 5850.500000 1154.705374 5034.0 5442.25 5850.5 6258.75 6667.0
    libafl_naive 58500 2.0 5042.000000 8.485281 5036.0 5039.00 5042.0 5045.00 5048.0
    libafl_weighted 58500 3.0 5046.000000 15.620499 5036.0 5037.00 5038.0 5051.00 5064.0
    libafl_random 58500 3.0 5039.333333 5.859465 5035.0 5036.00 5037.0 5041.50 5046.0
    libafl_explore 58500 3.0 4991.000000 86.810138 4891.0 4963.00 5035.0 5041.00 5047.0
    libafl_covaccount 58500 4.0 4964.500000 86.125877 4889.0 4890.50 4962.5 5036.50 5044.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

vorbis_decode_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libafl_fast 59400 2.0 1110.000000 56.568542 1070.0 1090.0 1110.0 1130.0 1150.0
    libafl_random 59400 3.0 1075.333333 32.868425 1039.0 1061.5 1084.0 1093.5 1103.0
    libafl_covaccount 59400 1.0 1080.000000 NaN 1080.0 1080.0 1080.0 1080.0 1080.0
    libafl_explore 59400 3.0 828.333333 459.235597 307.0 656.0 1005.0 1089.0 1173.0
    libafl_weighted 59400 3.0 873.000000 296.187441 700.0 702.0 704.0 959.5 1215.0
    libafl_naive 59400 2.0 610.000000 121.622366 524.0 567.0 610.0 653.0 696.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

wireshark_fuzzshark summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libafl_fast 9900 3.0 56116.000000 4090.360864 53110.0 53787.0 54464.0 57619.0 60774.0
    libafl_covaccount 9900 8.0 53453.750000 2397.910028 50417.0 51374.0 53190.0 55562.5 56927.0
    libafl_random 9900 5.0 53533.200000 2258.161132 51859.0 51954.0 53134.0 53326.0 57393.0
    libafl_naive 9900 3.0 43684.333333 1647.135190 42300.0 42773.5 43247.0 44376.5 45506.0
    libafl_weighted 9900 11.0 42310.636364 2301.184098 36820.0 41254.0 42412.0 43533.5 45490.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

woff2_convert_woff2ttf_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libafl_naive 59400 6.0 942.666667 67.298341 873.0 885.75 935.0 996.25 1027.0
    libafl_random 59400 2.0 929.500000 54.447222 891.0 910.25 929.5 948.75 968.0
    libafl_weighted 59400 4.0 928.250000 17.480942 912.0 917.25 924.5 935.50 952.0
    libafl_covaccount 59400 2.0 917.000000 22.627417 901.0 909.00 917.0 925.00 933.0
    libafl_explore 59400 3.0 905.000000 31.953091 869.0 892.50 916.0 923.00 930.0
    libafl_fast 59400 2.0 826.000000 164.048773 710.0 768.00 826.0 884.00 942.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

zlib_zlib_uncompress_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libafl_fast 58500 6.0 458.833333 7.756718 450.0 453.75 459.5 460.0 472.0
    libafl_naive 58500 5.0 458.000000 7.176350 448.0 455.00 458.0 462.0 467.0
    libafl_random 58500 2.0 456.000000 8.485281 450.0 453.00 456.0 459.0 462.0
    libafl_covaccount 58500 2.0 451.000000 0.000000 451.0 451.00 451.0 451.0 451.0
    libafl_explore 58500 4.0 454.750000 9.535023 449.0 449.75 450.5 455.5 469.0
    libafl_weighted 58500 5.0 450.600000 6.503845 444.0 447.00 449.0 452.0 461.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

experiment data

You can download the raw data for this report here.

Check out the documentation on how to create customized reports using this data. Also see some example Colab notebooks for doing custom analysis on the data here.

Experiment Description:

(None,)