FuzzBench: 2025-04-01-libafl-scheduling2 report

experiment summary

We show two different aggregate (cross-benchmark) rankings of fuzzers. The first is based on the average of per-benchmarks scores, where the score represents the percentage of the highest reached median code-coverage on a given benchmark (higher value is better). The second ranking shows the average rank of fuzzers, after we rank them on each benchmark according to their median reached code-covereges (lower value is better).
By avg. score
average normalized score
fuzzer
aflsmart 71.89
aflfast 71.24
eclipser 69.93
fairfuzz 69.28
centipede 56.33
aflplusplus 10.71
libafl_fast 10.71
libafl_random 10.71
libfuzzer 9.15
afl 7.64
honggfuzz 7.01
mopt 6.59
libafl_covaccount 3.57
libafl 3.31
By avg. rank
average rank
fuzzer
aflsmart 2.57
eclipser 2.79
aflfast 3.46
fairfuzz 3.54
centipede 3.82
aflplusplus 4.82
honggfuzz 5.18
libfuzzer 5.21
mopt 5.29
afl 5.36
libafl 5.36
libafl_fast 5.36
libafl_random 5.36
libafl_covaccount 5.54
  • Critical difference diagram
    The diagram visualizes the average rank of fuzzers (second ranking above) while showing the significance of the differences as well. What is considered a "critical difference" (CD) is based on the Friedman/Nemenyi post-hoc test. See more in the documentation.
    Note: If a fuzzer does not support all benchmarks, its ranking as shown in this diagram can be lower than it should be. So please check the list of supported benchmarks for the fuzzer(s) of your interest. The list could be specified in the fuzzer's README.md like this.
  • Median relative code-coverages on each benchmark

    Note: The relative coverage summary table shows the median relative performance of each fuzzer to the experiment maximum. Thus the highest relative performance may not be 100%.
    trial_relative_coverage = trial_coverage / experiment_max_coverage

      libafl_fast libafl_random libafl_covaccount aflplusplus honggfuzz libafl mopt eclipser libfuzzer aflsmart aflfast centipede fairfuzz afl
    FuzzerMedian 100.00 100.00 99.00 97.00 94.00 90.00 89.50 93.00 86.00 92.00 92.00 89.50 88.00 88.00
    FuzzerMean 100.00 100.00 99.00 96.33 94.00 90.00 89.50 87.85 82.33 80.48 80.17 78.83 77.52 68.67
    bloaty_fuzz_target nan nan nan nan nan nan nan 87.00 nan 87.00 87.00 nan 74.00 nan
    curl_curl_fuzzer_http nan nan nan nan nan nan nan 93.00 nan 93.00 92.00 nan 84.00 nan
    freetype2_ftfuzzer nan nan nan nan nan nan nan 53.00 nan 48.00 46.00 41.00 45.00 nan
    harfbuzz_hb-shape-fuzzer nan nan nan nan nan nan nan 97.00 nan 96.00 96.00 nan 84.00 nan
    jsoncpp_jsoncpp_fuzzer nan nan nan nan nan nan nan 98.00 nan 98.00 98.00 98.00 98.00 nan
    lcms_cms_transform_fuzzer nan nan nan nan nan nan nan 69.00 nan 37.00 26.00 44.00 50.00 nan
    libaom_av1_dec_fuzzer nan nan nan 95.00 93.00 nan nan nan 86.00 nan nan nan nan 88.00
    libjpeg-turbo_libjpeg_turbo_fuzzer nan nan nan nan nan nan nan nan nan 82.00 98.00 82.00 99.00 nan
    libpcap_fuzz_both nan nan nan nan nan nan nan 54.00 nan 0.00 0.00 67.00 0.00 nan
    libpng_libpng_read_fuzzer nan nan nan nan nan nan nan 98.00 nan 98.00 96.00 99.00 98.00 nan
    libvpx_vpx_dec_fuzzer_vp9 nan 100.00 nan nan nan nan nan nan nan nan nan nan nan nan
    libxml2_xml nan nan nan nan nan nan nan 98.00 nan 98.00 98.00 94.00 90.00 nan
    libxslt_xpath nan nan nan nan nan nan nan 97.00 nan 98.00 96.00 96.00 98.00 nan
    mbedtls_fuzz_dtlsclient nan nan nan nan nan nan nan 72.00 nan 72.00 70.00 70.00 75.00 nan
    openssl_x509 nan nan nan nan nan nan nan 99.00 nan 99.00 99.00 99.00 99.00 nan
    openthread_ot-ip6-send-fuzzer nan nan nan nan nan nan nan 97.00 nan 94.00 94.00 94.00 89.00 nan
    php_php-fuzz-execute nan nan nan 97.00 95.00 nan 89.00 89.00 69.00 88.00 90.00 nan 87.00 28.00
    php_php-fuzz-parser nan nan nan 97.00 nan 90.00 90.00 nan 92.00 90.00 90.00 nan 90.00 90.00
    php_php-fuzz-unserialize 100.00 nan nan nan nan nan nan nan nan nan nan nan nan nan
    php_php-fuzz-unserializehash 100.00 100.00 nan nan nan nan nan nan nan nan nan nan nan nan
    proj4_proj_crs_to_crs_fuzzer nan nan nan nan nan nan nan 86.00 nan 14.00 13.00 14.00 14.00 nan
    re2_fuzzer nan nan nan nan nan nan nan 99.00 nan 99.00 99.00 96.00 99.00 nan
    sqlite3_ossfuzz nan nan nan nan nan nan nan 92.00 nan 92.00 92.00 63.00 58.00 nan
    systemd_fuzz-link-parser nan nan nan nan nan nan nan 93.00 nan 93.00 93.00 100.00 88.00 nan
    vorbis_decode_fuzzer nan nan nan nan nan nan nan 90.00 nan 90.00 90.00 82.00 89.00 nan
    wireshark_fuzzshark 100.00 100.00 99.00 nan nan nan nan nan nan nan nan nan nan nan
    woff2_convert_woff2ttf_fuzzer nan nan nan nan nan nan nan nan nan 89.00 87.00 85.00 79.00 nan
    zlib_zlib_uncompress_fuzzer nan nan nan nan nan nan nan 96.00 nan 96.00 94.00 95.00 96.00 nan
    • Fuzzers are sorted by "FuzzerMean" (average median relative coverage), highest on the left.
    • Green background = highest relative median coverage.
    • Blue gradient background = greater than 95% relative median coverage.

bloaty_fuzz_target summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
error
The following fuzzers do not have enough samples: eclipser.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflsmart 82800 20.0 6146.000000 146.262561 5784.0 6070.75 6104.0 6271.00 6381.0
    eclipser 82800 11.0 6053.181818 68.930136 5914.0 6022.50 6078.0 6095.50 6135.0
    aflfast 82800 20.0 6077.150000 116.111233 5846.0 6016.00 6076.0 6140.25 6335.0
    fairfuzz 82800 16.0 5189.937500 67.159977 5111.0 5135.25 5170.0 5226.00 5323.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

curl_curl_fuzzer_http summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflsmart 82800 20.0 10385.000000 132.985951 10018.0 10363.25 10420.0 10455.25 10544.0
    eclipser 82800 17.0 10351.470588 163.547057 9942.0 10353.00 10381.0 10436.00 10545.0
    aflfast 82800 20.0 10300.900000 57.898187 10100.0 10277.00 10306.0 10340.75 10375.0
    fairfuzz 82800 19.0 9222.578947 391.462545 8215.0 8926.50 9379.0 9512.00 9714.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

freetype2_ftfuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
error
The following fuzzers do not have enough samples: centipede.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    eclipser 82800 18.0 9312.222222 98.567717 9066.0 9283.00 9317.0 9355.5 9516.0
    aflsmart 82800 20.0 8342.750000 174.588501 7870.0 8314.75 8393.0 8435.5 8535.0
    aflfast 82800 20.0 8059.000000 238.403903 7700.0 7852.50 8061.5 8289.0 8362.0
    fairfuzz 82800 17.0 7936.117647 236.373349 7764.0 7809.00 7863.0 7895.0 8582.0
    centipede 82800 14.0 7190.000000 120.755315 7001.0 7121.50 7192.0 7230.5 7434.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

harfbuzz_hb-shape-fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
error
The following fuzzers do not have enough samples: fairfuzz.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    eclipser 82800 20.0 10769.400000 51.530676 10623.0 10744.25 10777.0 10789.50 10868.0
    aflsmart 82800 20.0 10763.700000 42.521945 10652.0 10753.00 10764.5 10791.00 10839.0
    aflfast 82800 20.0 10677.750000 59.487172 10566.0 10633.00 10675.5 10714.50 10791.0
    fairfuzz 82800 14.0 9509.571429 347.646530 9071.0 9259.50 9399.0 9715.75 10155.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

jsoncpp_jsoncpp_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
error
The following fuzzers do not have enough samples: centipede.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    centipede 82800 12.0 520.166667 2.167249 518.0 518.00 519.5 522.00 524.0
    eclipser 82800 20.0 516.200000 5.176872 505.0 516.50 518.0 520.00 520.0
    aflfast 82800 20.0 517.000000 3.670652 502.0 517.00 517.0 519.00 519.0
    aflsmart 82800 19.0 516.105263 4.689169 503.0 517.00 517.0 519.00 520.0
    fairfuzz 82800 16.0 516.437500 2.988171 509.0 516.75 517.0 518.25 520.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

lcms_cms_transform_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
error
The following fuzzers do not have enough samples: centipede.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    eclipser 82800 17.0 1656.823529 137.802955 1398.0 1537.0 1671.0 1753.00 1893.0
    fairfuzz 82800 18.0 1274.277778 397.407600 800.0 901.0 1221.0 1636.25 1940.0
    centipede 82800 8.0 1077.750000 299.600567 777.0 789.0 1088.5 1361.00 1370.0
    aflsmart 82800 20.0 1074.750000 469.254714 650.0 652.0 900.0 1561.00 1791.0
    aflfast 82800 20.0 655.900000 152.566086 476.0 629.5 640.5 644.25 1276.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

libaom_av1_dec_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
error
The following fuzzers do not have enough samples: afl, honggfuzz, libfuzzer.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus 3600 20.0 10881.55 304.201885 10253.0 10806.50 10947.5 11163.00 11215.0
    honggfuzz 3600 10.0 10645.40 125.164070 10362.0 10618.25 10685.0 10701.25 10816.0
    afl 3600 10.0 10148.10 66.237703 10000.0 10124.75 10151.5 10171.25 10250.0
    libfuzzer 3600 10.0 9910.90 75.164191 9813.0 9839.25 9939.0 9962.25 10027.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

libjpeg-turbo_libjpeg_turbo_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
error
The following fuzzers do not have enough samples: fairfuzz, centipede.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    fairfuzz 82800 13.0 3072.615385 17.399897 3017.0 3072.0 3079.0 3080.0 3084.0
    aflfast 82800 20.0 3047.050000 30.163545 3007.0 3015.0 3053.5 3078.0 3081.0
    centipede 82800 7.0 2545.000000 2.708013 2541.0 2543.5 2546.0 2546.0 2549.0
    aflsmart 82800 20.0 2544.750000 1.773341 2543.0 2543.0 2545.0 2546.0 2549.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

libpcap_fuzz_both summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
error
The following fuzzers do not have enough samples: centipede, aflfast.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    centipede 82800 11.0 2759.090909 890.255183 101.0 2904.0 3019.0 3076.5 3227.0
    eclipser 82800 19.0 2461.052632 196.678947 1977.0 2403.0 2446.0 2609.0 2706.0
    aflfast 82800 6.0 38.500000 4.929503 34.0 34.0 38.5 43.0 43.0
    aflsmart 82800 20.0 34.000000 0.000000 34.0 34.0 34.0 34.0 34.0
    fairfuzz 82800 16.0 37.375000 4.500000 34.0 34.0 34.0 43.0 43.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

libpng_libpng_read_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    centipede 82800 17.0 2013.529412 3.710082 2006.0 2013.00 2015.0 2016.00 2018.0
    eclipser 82800 18.0 1983.000000 27.563830 1900.0 1988.25 1993.5 1996.25 1999.0
    aflsmart 82800 20.0 1964.400000 41.639303 1888.0 1926.50 1992.5 1995.25 1998.0
    fairfuzz 82800 18.0 1972.166667 30.020091 1889.0 1955.75 1981.0 1996.00 1999.0
    aflfast 82800 20.0 1940.050000 40.045336 1856.0 1926.25 1946.5 1973.50 1987.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

libvpx_vpx_dec_fuzzer_vp9 summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libafl_random 0 1.0 3648.0 NaN 3648.0 3648.0 3648.0 3648.0 3648.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row
  • Code coverage reports for each fuzzer on this benchmark
    libafl_random

libxml2_xml summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
error
The following fuzzers do not have enough samples: centipede.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    eclipser 82800 17.0 15366.176471 70.190130 15247.0 15310.0 15373.0 15407.00 15477.0
    aflsmart 82800 20.0 15356.900000 64.464515 15142.0 15341.5 15371.0 15390.50 15431.0
    aflfast 82800 20.0 15282.950000 77.642551 15097.0 15249.5 15282.5 15340.75 15388.0
    centipede 82800 11.0 14674.909091 108.888433 14450.0 14612.0 14706.0 14761.00 14787.0
    fairfuzz 82800 19.0 14034.842105 437.228934 12545.0 13923.0 14065.0 14210.00 14843.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

libxslt_xpath summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
error
The following fuzzers do not have enough samples: centipede.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    fairfuzz 82800 17.0 10804.470588 200.035346 10162.0 10772.00 10855.0 10906.00 11044.0
    aflsmart 82800 20.0 10844.850000 43.367797 10793.0 10816.50 10826.5 10859.25 10942.0
    eclipser 82800 18.0 10767.611111 123.770252 10424.0 10773.25 10801.5 10834.00 10888.0
    centipede 82800 12.0 10707.000000 103.919549 10550.0 10641.75 10686.0 10793.00 10863.0
    aflfast 82800 20.0 10631.400000 116.451662 10368.0 10570.25 10677.0 10705.50 10796.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

mbedtls_fuzz_dtlsclient summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
error
The following fuzzers do not have enough samples: fairfuzz, centipede.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    fairfuzz 82800 15.0 2849.466667 159.218119 2754.0 2779.00 2801.0 2807.50 3296.0
    aflsmart 82800 20.0 2709.000000 26.942434 2665.0 2701.75 2705.5 2713.00 2792.0
    eclipser 82800 16.0 2726.562500 280.074030 2496.0 2674.00 2692.5 2715.00 3730.0
    centipede 82800 14.0 2692.142857 193.889285 2610.0 2629.00 2640.5 2652.75 3363.0
    aflfast 82800 20.0 2561.850000 115.700737 2312.0 2580.00 2611.0 2629.75 2658.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

openssl_x509 summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
error
The following fuzzers do not have enough samples: eclipser, centipede.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflsmart 82800 20.0 5826.300000 5.582869 5808.0 5827.00 5828.0 5829.0 5831.0
    eclipser 82800 15.0 5825.266667 4.096456 5817.0 5823.50 5827.0 5828.0 5831.0
    aflfast 82800 20.0 5822.100000 7.496666 5797.0 5820.00 5823.5 5826.5 5830.0
    centipede 82800 12.0 5821.833333 4.895886 5809.0 5821.25 5823.5 5824.0 5829.0
    fairfuzz 82800 16.0 5821.125000 2.753785 5816.0 5819.50 5822.0 5823.0 5825.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

openthread_ot-ip6-send-fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
error
The following fuzzers do not have enough samples: centipede.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    eclipser 82800 16.0 2984.937500 66.946216 2895.0 2918.25 2999.5 3036.75 3073.0
    aflsmart 82800 20.0 2896.650000 46.006035 2828.0 2886.25 2907.0 2912.25 3025.0
    aflfast 82800 20.0 2888.250000 43.931616 2810.0 2865.75 2906.0 2911.75 2974.0
    centipede 82800 13.0 2855.615385 74.386982 2691.0 2796.00 2890.0 2897.00 2955.0
    fairfuzz 82800 19.0 2779.105263 65.278799 2676.0 2745.50 2764.0 2801.50 2912.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

php_php-fuzz-execute summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
error
The following fuzzers do not have enough samples: afl.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus 82800 19.0 196409.368421 2254.730588 192728.0 194587.00 196638.0 197543.00 201889.0
    honggfuzz 82800 20.0 193316.750000 2159.492822 187607.0 192570.50 193771.5 194848.25 196958.0
    aflfast 82800 19.0 182207.842105 3305.528505 177006.0 179489.00 181790.0 184742.00 188363.0
    mopt 82800 19.0 181869.842105 3824.293897 176838.0 178893.50 181349.0 185123.00 189226.0
    eclipser 82800 19.0 180038.315789 3035.254814 174865.0 177803.00 179984.0 181784.00 186682.0
    aflsmart 82800 18.0 179983.888889 6285.419794 162257.0 177256.00 179538.5 184231.50 189420.0
    fairfuzz 82800 19.0 172632.631579 10515.290862 153890.0 166195.00 175863.0 178701.50 184407.0
    libfuzzer 82800 19.0 140400.526316 1161.324597 139083.0 139494.50 140110.0 140958.50 143051.0
    afl 82800 14.0 57117.214286 1853.240205 54048.0 55851.75 56861.5 57807.50 60347.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

php_php-fuzz-parser summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
error
The following fuzzers do not have enough samples: aflplusplus, libfuzzer.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus 82800 2.0 17826.000000 571.342279 17422.0 17624.00 17826.0 18028.00 18230.0
    libfuzzer 82800 2.0 16793.500000 72.831998 16742.0 16767.75 16793.5 16819.25 16845.0
    libafl 82800 20.0 16512.800000 43.373045 16448.0 16482.75 16500.0 16530.00 16630.0
    fairfuzz 82800 19.0 16278.578947 282.829221 15765.0 16071.50 16473.0 16519.50 16541.0
    aflsmart 82800 19.0 16455.736842 29.858634 16401.0 16433.00 16468.0 16479.00 16490.0
    afl 82800 19.0 16462.789474 29.656513 16397.0 16442.00 16467.0 16478.00 16517.0
    mopt 82800 18.0 16427.777778 24.685208 16371.0 16415.50 16435.0 16448.25 16462.0
    aflfast 82800 20.0 16405.300000 51.390251 16212.0 16395.75 16410.0 16427.50 16473.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

php_php-fuzz-unserialize summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libafl_fast 0 1.0 25744.0 NaN 25744.0 25744.0 25744.0 25744.0 25744.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row
  • Code coverage reports for each fuzzer on this benchmark
    libafl_fast

php_php-fuzz-unserializehash summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libafl_fast 0 1.0 26668.0 NaN 26668.0 26668.0 26668.0 26668.0 26668.0
    libafl_random 0 1.0 26668.0 NaN 26668.0 26668.0 26668.0 26668.0 26668.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

proj4_proj_crs_to_crs_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
error
The following fuzzers do not have enough samples: centipede.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    eclipser 82800 20.0 4850.350000 238.740175 4471.0 4688.75 4879.5 4962.25 5520.0
    centipede 82800 12.0 822.916667 5.550730 817.0 818.50 822.0 826.50 834.0
    fairfuzz 82800 20.0 779.350000 98.588606 494.0 810.00 816.0 819.00 821.0
    aflsmart 82800 20.0 805.450000 17.101785 740.0 804.75 807.0 815.25 821.0
    aflfast 82800 20.0 738.800000 92.916828 488.0 727.75 740.5 806.25 817.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

re2_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
error
The following fuzzers do not have enough samples: centipede.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflsmart 82800 20.0 2863.850000 17.502707 2792.0 2863.75 2867.5 2870.50 2876.0
    eclipser 82800 17.0 2849.176471 37.480387 2746.0 2854.00 2867.0 2872.00 2877.0
    fairfuzz 82800 16.0 2849.875000 34.960692 2757.0 2858.75 2864.0 2866.00 2872.0
    aflfast 82800 20.0 2851.500000 30.130418 2778.0 2860.50 2862.5 2866.25 2869.0
    centipede 82800 10.0 2767.000000 23.489951 2733.0 2753.25 2762.5 2775.50 2811.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

sqlite3_ossfuzz summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
error
The following fuzzers do not have enough samples: centipede, fairfuzz.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflsmart 82800 20.0 18982.900000 237.661613 18306.0 18933.75 19022.0 19117.00 19303.0
    aflfast 82800 20.0 18918.450000 267.101494 18518.0 18677.25 18938.0 19154.00 19247.0
    eclipser 82800 16.0 18884.375000 348.908751 17767.0 18835.25 18909.5 19083.75 19306.0
    centipede 82800 12.0 13113.083333 482.740559 12453.0 12802.50 13043.5 13350.75 14071.0
    fairfuzz 82800 12.0 12516.333333 1555.113169 10873.0 11305.75 12095.0 13230.75 15546.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

vorbis_decode_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
error
The following fuzzers do not have enough samples: centipede, eclipser.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflsmart 82800 20.0 1244.100000 19.558011 1199.0 1247.00 1251.5 1254.00 1259.0
    aflfast 82800 20.0 1245.400000 19.491969 1183.0 1246.75 1251.0 1253.75 1258.0
    eclipser 82800 15.0 1248.133333 5.514483 1236.0 1245.00 1248.0 1252.50 1255.0
    fairfuzz 82800 18.0 1228.111111 28.060625 1175.0 1208.25 1238.0 1250.75 1257.0
    centipede 82800 15.0 1143.266667 14.901901 1118.0 1134.00 1144.0 1150.50 1170.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

wireshark_fuzzshark summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
error
The following fuzzers do not have enough samples: libafl_fast, libafl_random.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libafl_fast 0 1.0 95877.0 NaN 95877.0 95877.00 95877.0 95877.00 95877.0
    libafl_random 0 1.0 95877.0 NaN 95877.0 95877.00 95877.0 95877.00 95877.0
    libafl_covaccount 0 4.0 95875.0 0.816497 95874.0 95874.75 95875.0 95875.25 95876.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

woff2_convert_woff2ttf_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
error
The following fuzzers do not have enough samples: centipede, aflsmart.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflsmart 82800 13.0 1105.461538 29.809438 1047.0 1090.00 1123.0 1127.00 1129.0
    aflfast 82800 20.0 1084.450000 26.198584 1032.0 1065.25 1090.5 1106.25 1124.0
    centipede 82800 15.0 1072.800000 15.279304 1036.0 1066.50 1069.0 1082.00 1098.0
    fairfuzz 82800 17.0 994.941176 30.150602 959.0 982.00 991.0 1001.00 1098.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

zlib_zlib_uncompress_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
error
The following fuzzers do not have enough samples: centipede, aflsmart.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    fairfuzz 82800 16.0 457.875000 3.685557 455.0 455.75 457.0 459.0 470.0
    aflsmart 82800 12.0 455.750000 13.948770 416.0 455.75 456.0 462.0 470.0
    eclipser 82800 16.0 451.687500 13.189484 423.0 450.25 455.0 458.0 471.0
    centipede 82800 13.0 454.153846 2.544476 451.0 452.00 454.0 456.0 459.0
    aflfast 82800 20.0 448.100000 15.109774 386.0 449.00 449.0 454.0 460.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

experiment data

You can download the raw data for this report here.

Check out the documentation on how to create customized reports using this data. Also see some example Colab notebooks for doing custom analysis on the data here.

Experiment Description:

(None,)