FuzzBench: 2023-09-04-libafl-1 report

experiment summary

We show two different aggregate (cross-benchmark) rankings of fuzzers. The first is based on the average of per-benchmarks scores, where the score represents the percentage of the highest reached median code-coverage on a given benchmark (higher value is better). The second ranking shows the average rank of fuzzers, after we rank them on each benchmark according to their median reached code-covereges (lower value is better).
By avg. score
average normalized score
fuzzer
libafl_fuzzbench_cmplog 79.87
libafl_fuzzbench_fast 78.94
libafl_fuzzbench_explore 78.94
libafl_fuzzbench_mopt 77.14
aflplusplus 76.04
libafl_fuzzbench_cov_accounting 75.97
libfuzzer 17.42
honggfuzz 11.76
afl 10.47
centipede 0.32
aflfast 0.00
aflsmart 0.00
eclipser 0.00
fairfuzz 0.00
mopt 0.00
By avg. rank
average rank
fuzzer
aflplusplus 2.24
libafl_fuzzbench_fast 2.24
libafl_fuzzbench_explore 2.29
libafl_fuzzbench_mopt 3.29
libafl_fuzzbench_cmplog 3.41
libafl_fuzzbench_cov_accounting 4.18
honggfuzz 5.88
libfuzzer 6.06
afl 6.24
centipede 6.29
aflfast 6.35
aflsmart 6.35
eclipser 6.35
fairfuzz 6.35
mopt 6.35
  • Critical difference diagram
    The diagram visualizes the average rank of fuzzers (second ranking above) while showing the significance of the differences as well. What is considered a "critical difference" (CD) is based on the Friedman/Nemenyi post-hoc test. See more in the documentation.
    Note: If a fuzzer does not support all benchmarks, its ranking as shown in this diagram can be lower than it should be. So please check the list of supported benchmarks for the fuzzer(s) of your interest. The list could be specified in the fuzzer's README.md like this.
  • Median relative code-coverages on each benchmark

    Note: The relative coverage summary table shows the median relative performance of each fuzzer to the experiment maximum. Thus the highest relative performance may not be 100%.
    trial_relative_coverage = trial_coverage / experiment_max_coverage

      libafl_fuzzbench_cmplog aflplusplus libafl_fuzzbench_explore libafl_fuzzbench_fast libafl_fuzzbench_mopt libafl_fuzzbench_cov_accounting libfuzzer honggfuzz afl centipede aflfast aflsmart eclipser fairfuzz mopt
    FuzzerMedian 97.00 99.00 99.00 99.00 98.00 97.00 95.00 48.00 47.00 5.00 0.00 0.00 0.00 0.00 0.00
    FuzzerMean 87.73 87.43 87.27 87.27 85.47 84.13 80.67 48.67 41.67 5.00 0.00 0.00 0.00 0.00 0.00
    arduinojson_json_fuzzer 99.00 100.00 99.00 99.00 99.00 99.00 nan nan nan nan nan nan nan nan nan
    assimp_assimp_fuzzer 78.00 nan 43.00 43.00 38.00 37.00 nan nan nan nan nan nan nan nan nan
    astc-encoder_fuzz_astc_physical_to_symbolic 95.00 nan 95.00 95.00 95.00 95.00 nan nan nan nan nan nan nan nan nan
    botan_tls_server nan 48.00 nan nan nan nan 48.00 48.00 47.00 nan nan nan nan nan nan
    brotli_decode_fuzzer 98.00 nan 99.00 99.00 99.00 99.00 nan nan nan nan nan nan nan nan nan
    double-conversion_string_to_double_fuzzer 97.00 99.00 97.00 97.00 97.00 97.00 nan nan nan nan nan nan nan nan nan
    draco_draco_pc_decoder_fuzzer 74.00 90.00 93.00 93.00 73.00 60.00 nan nan nan nan nan nan nan nan nan
    dropbear_fuzzer-postauth_nomaths nan 96.00 nan nan nan nan 95.00 98.00 78.00 nan nan nan nan nan nan
    firestore_firestore_serializer_fuzzer 100.00 100.00 100.00 100.00 100.00 99.00 nan nan nan nan nan nan nan nan nan
    fmt_chrono-duration-fuzzer 98.00 99.00 99.00 99.00 98.00 99.00 nan nan nan nan nan nan nan nan nan
    guetzli_guetzli_fuzzer 97.00 99.00 99.00 99.00 99.00 97.00 nan nan nan nan nan nan nan nan nan
    icu_unicode_string_codepage_create_fuzzer 99.00 99.00 99.00 99.00 99.00 99.00 nan nan nan nan nan nan nan nan nan
    jansson_json_load_dump_fuzzer 99.00 99.00 99.00 99.00 99.00 99.00 nan nan nan nan nan nan nan nan nan
    libaom_av1_dec_fuzzer 92.00 97.00 98.00 98.00 97.00 93.00 nan nan nan nan nan nan nan nan nan
    libcoap_pdu_parse_fuzzer 91.00 99.00 90.00 90.00 90.00 90.00 nan nan nan nan nan nan nan nan nan
    libhevc_hevc_dec_fuzzer 99.00 99.00 99.00 99.00 99.00 99.00 nan nan nan 5.00 0.00 0.00 0.00 0.00 0.00
    librdkafka_fuzz_regex 0.00 0.00 0.00 0.00 0.00 0.00 99.00 0.00 0.00 nan nan nan nan nan nan
    • Fuzzers are sorted by "FuzzerMean" (average median relative coverage), highest on the left.
    • Green background = highest relative median coverage.
    • Blue gradient background = greater than 95% relative median coverage.

arduinojson_json_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus 82800 20.0 401.60 0.502625 401.0 401.0 402.0 402.00 402.0
    libafl_fuzzbench_cmplog 82800 20.0 399.25 0.910465 397.0 399.0 399.0 400.00 401.0
    libafl_fuzzbench_explore 82800 20.0 398.65 0.933302 397.0 398.0 399.0 399.00 400.0
    libafl_fuzzbench_fast 82800 20.0 398.90 0.967906 397.0 398.0 399.0 399.25 401.0
    libafl_fuzzbench_mopt 82800 20.0 398.75 0.966546 397.0 398.0 399.0 399.00 400.0
    libafl_fuzzbench_cov_accounting 82800 20.0 398.45 0.887041 397.0 398.0 398.5 399.00 400.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

assimp_assimp_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
error
The following fuzzers do not have enough samples: libafl_fuzzbench_cmplog.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libafl_fuzzbench_cmplog 7200 7.0 4881.571429 158.684023 4719.0 4778.50 4870.0 4913.0 5199.0
    libafl_fuzzbench_explore 7200 16.0 2769.125000 337.017680 2345.0 2512.25 2695.0 2919.5 3532.0
    libafl_fuzzbench_fast 7200 17.0 2837.882353 394.249585 2409.0 2479.00 2670.0 3051.0 3512.0
    libafl_fuzzbench_mopt 7200 14.0 2491.357143 228.584139 2203.0 2298.75 2394.5 2699.5 2841.0
    libafl_fuzzbench_cov_accounting 7200 16.0 2439.625000 222.585077 2142.0 2254.75 2339.0 2622.5 2822.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

astc-encoder_fuzz_astc_physical_to_symbolic summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libafl_fuzzbench_cmplog 82800 20.0 489.60 1.667018 488.0 489.00 489.0 490.00 495.0
    libafl_fuzzbench_cov_accounting 82800 20.0 489.30 1.341641 488.0 488.75 489.0 489.25 493.0
    libafl_fuzzbench_explore 82800 20.0 488.95 0.825578 488.0 488.00 489.0 489.00 491.0
    libafl_fuzzbench_fast 82800 20.0 489.10 0.967906 488.0 488.75 489.0 489.25 492.0
    libafl_fuzzbench_mopt 82800 20.0 489.10 0.967906 488.0 488.75 489.0 489.00 491.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

botan_tls_server summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
error
The following fuzzers do not have enough samples: afl, honggfuzz, libfuzzer.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    honggfuzz 3600 10.0 1365.1 188.481328 1171.0 1242.75 1252.5 1549.00 1671.0
    aflplusplus 3600 20.0 1333.2 203.259906 1179.0 1224.00 1252.0 1288.75 1792.0
    libfuzzer 3600 10.0 1217.0 34.062035 1176.0 1178.25 1241.5 1242.00 1248.0
    afl 3600 10.0 1239.9 6.172520 1235.0 1236.25 1237.5 1241.25 1255.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

brotli_decode_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libafl_fuzzbench_explore 82800 20.0 901.35 1.348488 899.0 900.00 902.0 902.00 903.0
    libafl_fuzzbench_fast 82800 20.0 902.00 1.414214 899.0 901.75 902.0 902.25 904.0
    libafl_fuzzbench_mopt 82800 20.0 901.40 2.663595 893.0 900.75 901.5 904.00 904.0
    libafl_fuzzbench_cov_accounting 82800 20.0 896.30 4.995788 887.0 892.75 896.0 901.00 903.0
    libafl_fuzzbench_cmplog 82800 20.0 892.30 3.585057 889.0 890.00 891.0 892.00 901.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

double-conversion_string_to_double_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus 82800 20.0 507.75 1.831738 504.0 506.00 508.0 509.00 510.0
    libafl_fuzzbench_cmplog 82800 20.0 497.90 2.221901 494.0 496.75 498.0 499.25 502.0
    libafl_fuzzbench_cov_accounting 82800 20.0 497.90 1.889026 493.0 497.00 498.0 499.00 502.0
    libafl_fuzzbench_explore 82800 20.0 497.20 2.504732 491.0 495.75 498.0 499.00 500.0
    libafl_fuzzbench_fast 82800 20.0 497.60 1.875044 494.0 496.00 498.0 499.00 501.0
    libafl_fuzzbench_mopt 82800 20.0 497.25 1.970172 494.0 495.75 498.0 499.00 500.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

draco_draco_pc_decoder_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
error
The following fuzzers do not have enough samples: aflplusplus.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libafl_fuzzbench_fast 82800 20.0 1902.750000 77.139569 1706.0 1867.00 1913.5 1945.25 2036.0
    libafl_fuzzbench_explore 82800 20.0 1888.400000 81.994480 1684.0 1876.75 1909.5 1937.75 2003.0
    aflplusplus 82800 15.0 1850.066667 96.068775 1698.0 1779.00 1839.0 1931.50 1996.0
    libafl_fuzzbench_cmplog 82800 20.0 1501.350000 116.950406 1266.0 1387.75 1521.0 1602.25 1630.0
    libafl_fuzzbench_mopt 82800 20.0 1479.550000 167.750276 1114.0 1388.75 1489.5 1590.00 1807.0
    libafl_fuzzbench_cov_accounting 82800 20.0 1230.950000 138.284404 1006.0 1138.50 1230.0 1309.75 1491.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

dropbear_fuzzer-postauth_nomaths summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    honggfuzz 3600 10.0 883.6 9.570789 860.0 881.25 885.5 889.0 895.0
    aflplusplus 3600 10.0 863.0 2.538591 858.0 861.00 864.5 865.0 865.0
    libfuzzer 3600 10.0 857.1 5.877452 841.0 857.00 858.5 860.0 861.0
    afl 3600 10.0 701.0 0.000000 701.0 701.00 701.0 701.0 701.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

firestore_firestore_serializer_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus 82800 20.0 287.0 0.000000 287.0 287.0 287.0 287.0 287.0
    libafl_fuzzbench_cmplog 82800 20.0 286.3 0.978721 285.0 285.0 287.0 287.0 287.0
    libafl_fuzzbench_explore 82800 20.0 286.3 0.978721 285.0 285.0 287.0 287.0 287.0
    libafl_fuzzbench_fast 82800 20.0 286.3 0.978721 285.0 285.0 287.0 287.0 287.0
    libafl_fuzzbench_mopt 82800 20.0 286.5 0.888523 285.0 286.5 287.0 287.0 287.0
    libafl_fuzzbench_cov_accounting 82800 20.0 286.0 1.025978 285.0 285.0 286.0 287.0 287.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

fmt_chrono-duration-fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus 82800 20.0 1089.80 3.607011 1081.0 1086.75 1091.0 1093.00 1095.0
    libafl_fuzzbench_explore 82800 20.0 1086.85 3.375999 1082.0 1083.75 1087.0 1089.25 1092.0
    libafl_fuzzbench_fast 82800 20.0 1087.00 4.192726 1080.0 1083.75 1087.0 1090.00 1094.0
    libafl_fuzzbench_cov_accounting 82800 20.0 1086.30 3.921600 1078.0 1084.50 1086.5 1088.00 1095.0
    libafl_fuzzbench_mopt 82800 20.0 1081.35 4.648429 1075.0 1077.75 1080.0 1084.25 1091.0
    libafl_fuzzbench_cmplog 82800 20.0 1080.10 4.315212 1071.0 1078.00 1079.5 1083.25 1090.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

guetzli_guetzli_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus 82800 20.0 1498.30 5.391026 1484.0 1495.00 1498.5 1502.25 1506.0
    libafl_fuzzbench_mopt 82800 20.0 1486.85 8.755600 1473.0 1475.75 1491.5 1492.25 1498.0
    libafl_fuzzbench_explore 82800 20.0 1486.35 8.405982 1472.0 1480.00 1491.0 1492.00 1496.0
    libafl_fuzzbench_fast 82800 20.0 1487.75 8.403477 1469.0 1488.75 1491.0 1492.00 1497.0
    libafl_fuzzbench_cmplog 82800 20.0 1478.40 8.604772 1471.0 1472.75 1474.0 1480.75 1496.0
    libafl_fuzzbench_cov_accounting 82800 20.0 1475.20 12.120448 1460.0 1465.00 1472.5 1487.50 1493.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

icu_unicode_string_codepage_create_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus 82800 20.0 1339.85 0.745160 1339.0 1339.00 1340.0 1340.00 1341.0
    libafl_fuzzbench_cmplog 82800 20.0 1331.00 16.186414 1298.0 1337.75 1339.0 1339.00 1340.0
    libafl_fuzzbench_explore 82800 20.0 1336.85 8.731522 1300.0 1338.00 1339.0 1339.25 1340.0
    libafl_fuzzbench_fast 82800 20.0 1337.10 9.008183 1299.0 1338.00 1339.0 1340.00 1341.0
    libafl_fuzzbench_cov_accounting 82800 20.0 1330.00 15.610388 1299.0 1330.75 1338.5 1339.00 1341.0
    libafl_fuzzbench_mopt 82800 20.0 1330.65 14.042323 1300.0 1330.25 1338.0 1339.00 1340.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

jansson_json_load_dump_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus 82800 20.0 767.80 1.472556 765.0 767.00 768.0 769.0 770.0
    libafl_fuzzbench_cov_accounting 82800 20.0 767.25 1.446411 764.0 766.75 768.0 768.0 770.0
    libafl_fuzzbench_cmplog 82800 20.0 767.55 1.503505 765.0 767.00 767.5 769.0 770.0
    libafl_fuzzbench_explore 82800 20.0 767.30 1.260743 765.0 767.00 767.0 767.5 770.0
    libafl_fuzzbench_fast 82800 20.0 766.75 1.517442 764.0 766.00 767.0 768.0 769.0
    libafl_fuzzbench_mopt 82800 20.0 766.65 1.598519 764.0 765.75 767.0 768.0 770.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

libaom_av1_dec_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libafl_fuzzbench_fast 82800 20.0 11187.25 116.294940 10982.0 11086.00 11229.5 11277.50 11348.0
    libafl_fuzzbench_explore 82800 20.0 11167.55 150.153422 10872.0 11075.50 11193.0 11298.50 11362.0
    aflplusplus 82800 20.0 11048.45 295.341457 10282.0 10997.75 11104.5 11242.25 11352.0
    libafl_fuzzbench_mopt 82800 20.0 10981.50 162.011858 10556.0 10958.00 11027.5 11085.25 11202.0
    libafl_fuzzbench_cov_accounting 82800 20.0 10622.05 209.562571 10273.0 10435.75 10635.0 10714.50 11027.0
    libafl_fuzzbench_cmplog 82800 20.0 10569.15 169.919261 10356.0 10411.50 10545.5 10675.00 10905.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

libcoap_pdu_parse_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus 82800 20.0 817.35 1.268028 815.0 817.00 817.0 818.0 821.0
    libafl_fuzzbench_cmplog 82800 20.0 776.75 34.546498 742.0 746.00 750.5 813.0 816.0
    libafl_fuzzbench_cov_accounting 82800 20.0 745.15 1.182103 743.0 744.00 745.0 746.0 747.0
    libafl_fuzzbench_explore 82800 20.0 745.25 1.251315 743.0 744.75 745.0 746.0 747.0
    libafl_fuzzbench_fast 82800 20.0 745.50 1.277333 743.0 745.00 745.0 746.0 749.0
    libafl_fuzzbench_mopt 82800 20.0 744.50 1.147079 743.0 743.75 745.0 745.0 747.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

libhevc_hevc_dec_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus 82800 20.0 10332.70 47.309841 10219.0 10331.50 10355.0 10360.00 10375.0
    libafl_fuzzbench_explore 82800 20.0 10324.55 7.096886 10311.0 10320.50 10323.0 10330.00 10337.0
    libafl_fuzzbench_fast 82800 20.0 10323.40 8.851999 10307.0 10317.00 10323.0 10330.25 10342.0
    libafl_fuzzbench_mopt 82800 20.0 10316.95 10.865082 10289.0 10308.00 10319.5 10324.25 10334.0
    libafl_fuzzbench_cov_accounting 82800 20.0 10303.55 14.365437 10273.0 10299.00 10306.0 10309.25 10334.0
    libafl_fuzzbench_cmplog 82800 20.0 10300.95 11.422393 10281.0 10292.75 10298.0 10311.25 10321.0
    centipede 82800 20.0 1260.30 1723.585892 555.0 555.00 557.0 557.00 5651.0
    aflfast 82800 20.0 0.00 0.000000 0.0 0.00 0.0 0.00 0.0
    aflsmart 82800 20.0 0.00 0.000000 0.0 0.00 0.0 0.00 0.0
    eclipser 82800 17.0 0.00 0.000000 0.0 0.00 0.0 0.00 0.0
    fairfuzz 82800 20.0 0.00 0.000000 0.0 0.00 0.0 0.00 0.0
    mopt 82800 20.0 0.00 0.000000 0.0 0.00 0.0 0.00 0.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

librdkafka_fuzz_regex summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
error
The following fuzzers do not have enough samples: afl, aflplusplus, honggfuzz, libfuzzer.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libfuzzer 900 10.0 379.8 4.661902 367.0 381.0 381.5 382.0 382.0
    afl 900 10.0 148.0 191.110788 0.0 0.0 0.0 367.0 378.0
    aflplusplus 900 10.0 73.4 154.752275 0.0 0.0 0.0 0.0 371.0
    honggfuzz 900 10.0 68.1 143.576886 0.0 0.0 0.0 0.0 344.0
    libafl_fuzzbench_cmplog 900 20.0 19.0 84.970583 0.0 0.0 0.0 0.0 380.0
    libafl_fuzzbench_cov_accounting 900 20.0 0.0 0.000000 0.0 0.0 0.0 0.0 0.0
    libafl_fuzzbench_explore 900 20.0 0.0 0.000000 0.0 0.0 0.0 0.0 0.0
    libafl_fuzzbench_fast 900 20.0 0.0 0.000000 0.0 0.0 0.0 0.0 0.0
    libafl_fuzzbench_mopt 900 20.0 18.9 84.523370 0.0 0.0 0.0 0.0 378.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

experiment data

You can download the raw data for this report here.

Check out the documentation on how to create customized reports using this data. Also see some example Colab notebooks for doing custom analysis on the data here.

Experiment Description:

(None,)