FuzzBench: 2023-09-18-libafl report

experiment summary

We show two different aggregate (cross-benchmark) rankings of fuzzers. The first is based on the average of per-benchmarks scores, where the score represents the percentage of the highest reached median code-coverage on a given benchmark (higher value is better). The second ranking shows the average rank of fuzzers, after we rank them on each benchmark according to their median reached code-covereges (lower value is better).
By avg. score
average normalized score
fuzzer
aflplusplus 97.49
libafl_fuzzbench_grimoire 68.61
libfuzzer 37.98
honggfuzz 28.15
afl 27.30
centipede 0.32
aflfast 0.00
aflsmart 0.00
eclipser 0.00
fairfuzz 0.00
mopt 0.00
By avg. rank
average rank
fuzzer
aflplusplus 1.18
libfuzzer 2.76
honggfuzz 2.94
libafl_fuzzbench_grimoire 2.94
afl 3.12
centipede 3.82
aflfast 3.88
aflsmart 3.88
eclipser 3.88
fairfuzz 3.88
mopt 3.88
  • Critical difference diagram
    The diagram visualizes the average rank of fuzzers (second ranking above) while showing the significance of the differences as well. What is considered a "critical difference" (CD) is based on the Friedman/Nemenyi post-hoc test. See more in the documentation.
    Note: If a fuzzer does not support all benchmarks, its ranking as shown in this diagram can be lower than it should be. So please check the list of supported benchmarks for the fuzzer(s) of your interest. The list could be specified in the fuzzer's README.md like this.
  • Median relative code-coverages on each benchmark

    Note: The relative coverage summary table shows the median relative performance of each fuzzer to the experiment maximum. Thus the highest relative performance may not be 100%.
    trial_relative_coverage = trial_coverage / experiment_max_coverage

      libafl_fuzzbench_grimoire aflplusplus libfuzzer honggfuzz afl centipede aflfast aflsmart eclipser fairfuzz mopt
    FuzzerMedian 97.50 99.00 95.00 68.50 50.00 5.00 0.00 0.00 0.00 0.00 0.00
    FuzzerMean 96.25 90.59 78.14 61.67 57.14 5.00 0.00 0.00 0.00 0.00 0.00
    arduinojson_json_fuzzer 99.00 100.00 nan nan nan nan nan nan nan nan nan
    assimp_assimp_fuzzer nan 64.00 60.00 87.00 50.00 nan nan nan nan nan nan
    astc-encoder_fuzz_astc_physical_to_symbolic 95.00 99.00 99.00 79.00 99.00 nan nan nan nan nan nan
    botan_tls_server nan 48.00 48.00 48.00 47.00 nan nan nan nan nan nan
    brotli_decode_fuzzer 97.00 99.00 98.00 nan 98.00 nan nan nan nan nan nan
    double-conversion_string_to_double_fuzzer 97.00 99.00 nan nan nan nan nan nan nan nan nan
    draco_draco_pc_decoder_fuzzer nan 59.00 48.00 58.00 0.00 nan nan nan nan nan nan
    dropbear_fuzzer-postauth_nomaths nan 96.00 95.00 98.00 78.00 nan nan nan nan nan nan
    firestore_firestore_serializer_fuzzer 100.00 100.00 nan nan nan nan nan nan nan nan nan
    fmt_chrono-duration-fuzzer 98.00 99.00 nan nan nan nan nan nan nan nan nan
    guetzli_guetzli_fuzzer 98.00 99.00 nan nan nan nan nan nan nan nan nan
    icu_unicode_string_codepage_create_fuzzer 96.00 99.00 nan nan nan nan nan nan nan nan nan
    jansson_json_load_dump_fuzzer 99.00 99.00 nan nan nan nan nan nan nan nan nan
    libaom_av1_dec_fuzzer 87.00 97.00 nan nan nan nan nan nan nan nan nan
    libcoap_pdu_parse_fuzzer 90.00 99.00 nan nan nan nan nan nan nan nan nan
    libhevc_hevc_dec_fuzzer 99.00 99.00 nan nan nan 5.00 0.00 0.00 0.00 0.00 0.00
    librdkafka_fuzz_regex nan 85.00 99.00 0.00 28.00 nan nan nan nan nan nan
    • Fuzzers are sorted by "FuzzerMean" (average median relative coverage), highest on the left.
    • Green background = highest relative median coverage.
    • Blue gradient background = greater than 95% relative median coverage.

arduinojson_json_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus 82800 20.0 401.6 0.502625 401.0 401.0 402.0 402.0 402.0
    libafl_fuzzbench_grimoire 82800 16.0 399.0 1.211060 397.0 398.0 399.0 400.0 402.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

assimp_assimp_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    honggfuzz 3600 10.0 3133.0 280.689231 2692.0 2949.00 3100.0 3280.00 3559.0
    aflplusplus 3600 10.0 2257.2 253.820934 1829.0 2128.75 2307.5 2440.75 2558.0
    libfuzzer 3600 10.0 2134.0 262.037741 1484.0 2086.00 2159.5 2302.00 2423.0
    afl 3600 10.0 1828.5 108.020831 1624.0 1788.50 1813.5 1899.25 1983.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

astc-encoder_fuzz_astc_physical_to_symbolic summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
error
The following fuzzers do not have enough samples: afl, aflplusplus, honggfuzz, libfuzzer.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 3600 10.0 511.6 0.699206 511.0 511.00 511.5 512.0 513.0
    aflplusplus 3600 10.0 511.2 0.632456 511.0 511.00 511.0 511.0 513.0
    libfuzzer 3600 10.0 511.0 0.000000 511.0 511.00 511.0 511.0 511.0
    libafl_fuzzbench_grimoire 3600 18.0 489.5 1.947849 488.0 489.00 489.0 489.0 497.0
    honggfuzz 3600 10.0 414.5 14.416426 406.0 406.75 409.0 409.0 448.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

botan_tls_server summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
error
The following fuzzers do not have enough samples: afl, honggfuzz, libfuzzer.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    honggfuzz 3600 10.0 1365.1 188.481328 1171.0 1242.75 1252.5 1549.00 1671.0
    aflplusplus 3600 20.0 1333.2 203.259906 1179.0 1224.00 1252.0 1288.75 1792.0
    libfuzzer 3600 10.0 1217.0 34.062035 1176.0 1178.25 1241.5 1242.00 1248.0
    afl 3600 10.0 1239.9 6.172520 1235.0 1236.25 1237.5 1241.25 1255.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

brotli_decode_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
error
The following fuzzers do not have enough samples: afl, aflplusplus, libfuzzer.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus 3600 10.0 898.200000 3.326660 894.0 894.50 900.0 900.0 902.0
    afl 3600 10.0 888.500000 6.485025 876.0 883.75 890.5 892.5 897.0
    libfuzzer 3600 10.0 889.300000 6.342975 878.0 889.25 890.5 891.0 897.0
    libafl_fuzzbench_grimoire 3600 17.0 881.705882 6.621600 875.0 877.00 878.0 889.0 895.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

double-conversion_string_to_double_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus 82800 20.0 507.75 1.831738 504.0 506.0 508.0 509.0 510.0
    libafl_fuzzbench_grimoire 82800 18.0 498.00 2.400980 495.0 496.0 498.0 500.0 502.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

draco_draco_pc_decoder_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
error
The following fuzzers do not have enough samples: afl, honggfuzz, libfuzzer.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus 3600 20.0 1187.7 174.849740 935.0 1036.00 1193.0 1339.50 1495.0
    honggfuzz 3600 10.0 1165.0 78.646890 1005.0 1119.75 1175.5 1227.00 1263.0
    libfuzzer 3600 10.0 950.3 93.641218 805.0 893.50 967.5 1026.25 1051.0
    afl 3600 10.0 7.0 0.000000 7.0 7.00 7.0 7.00 7.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

dropbear_fuzzer-postauth_nomaths summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    honggfuzz 3600 10.0 883.6 9.570789 860.0 881.25 885.5 889.0 895.0
    aflplusplus 3600 10.0 863.0 2.538591 858.0 861.00 864.5 865.0 865.0
    libfuzzer 3600 10.0 857.1 5.877452 841.0 857.00 858.5 860.0 861.0
    afl 3600 10.0 701.0 0.000000 701.0 701.00 701.0 701.0 701.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

firestore_firestore_serializer_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus 82800 20.0 287.000000 0.000000 287.0 287.0 287.0 287.0 287.0
    libafl_fuzzbench_grimoire 82800 18.0 286.666667 0.766965 285.0 287.0 287.0 287.0 287.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

fmt_chrono-duration-fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus 82800 20.0 1089.800000 3.607011 1081.0 1086.75 1091.0 1093.0 1095.0
    libafl_fuzzbench_grimoire 82800 17.0 1078.705882 4.412549 1072.0 1075.00 1078.0 1081.0 1087.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

guetzli_guetzli_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus 82800 20.0 1498.3000 5.391026 1484.0 1495.0 1498.5 1502.25 1506.0
    libafl_fuzzbench_grimoire 82800 16.0 1485.0625 8.812633 1472.0 1475.0 1489.5 1492.00 1496.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

icu_unicode_string_codepage_create_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
error
The following fuzzers do not have enough samples: libafl_fuzzbench_grimoire.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus 82800 20.0 1339.850000 0.745160 1339.0 1339.0 1340.0 1340.00 1341.0
    libafl_fuzzbench_grimoire 82800 14.0 1313.642857 19.325579 1298.0 1300.0 1300.0 1337.75 1340.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

jansson_json_load_dump_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus 82800 20.0 767.800000 1.472556 765.0 767.0 768.0 769.0 770.0
    libafl_fuzzbench_grimoire 82800 17.0 767.470588 1.328422 765.0 767.0 768.0 768.0 770.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

libaom_av1_dec_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus 82800 20.0 11048.450000 295.341457 10282.0 10997.75 11104.5 11242.25 11352.0
    libafl_fuzzbench_grimoire 82800 17.0 9948.882353 57.422646 9849.0 9919.00 9948.0 9972.00 10075.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

libcoap_pdu_parse_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus 82800 20.0 817.350000 1.268028 815.0 817.0 817.0 818.0 821.0
    libafl_fuzzbench_grimoire 82800 18.0 750.444444 16.143555 744.0 746.0 747.0 747.0 815.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

libhevc_hevc_dec_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus 82800 20.0 10332.700000 47.309841 10219.0 10331.5 10355.0 10360.0 10375.0
    libafl_fuzzbench_grimoire 82800 17.0 10309.529412 10.589604 10295.0 10300.0 10308.0 10317.0 10325.0
    centipede 82800 20.0 1260.300000 1723.585892 555.0 555.0 557.0 557.0 5651.0
    aflfast 82800 20.0 0.000000 0.000000 0.0 0.0 0.0 0.0 0.0
    aflsmart 82800 20.0 0.000000 0.000000 0.0 0.0 0.0 0.0 0.0
    eclipser 82800 17.0 0.000000 0.000000 0.0 0.0 0.0 0.0 0.0
    fairfuzz 82800 20.0 0.000000 0.000000 0.0 0.0 0.0 0.0 0.0
    mopt 82800 20.0 0.000000 0.000000 0.0 0.0 0.0 0.0 0.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

librdkafka_fuzz_regex summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libfuzzer 3600 10.0 379.8 4.661902 367.0 381.00 381.5 382.00 382.0
    aflplusplus 3600 10.0 288.4 110.639756 0.0 266.25 325.0 350.00 371.0
    afl 3600 10.0 169.6 184.713712 0.0 0.00 107.5 367.75 378.0
    honggfuzz 3600 10.0 83.4 143.665661 0.0 0.00 0.0 114.75 344.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

experiment data

You can download the raw data for this report here.

Check out the documentation on how to create customized reports using this data. Also see some example Colab notebooks for doing custom analysis on the data here.

Experiment Description:

(None,)