FuzzBench: 2023-09-05-libafl-3 report

experiment summary

We show two different aggregate (cross-benchmark) rankings of fuzzers. The first is based on the average of per-benchmarks scores, where the score represents the percentage of the highest reached median code-coverage on a given benchmark (higher value is better). The second ranking shows the average rank of fuzzers, after we rank them on each benchmark according to their median reached code-covereges (lower value is better).
By avg. score
average normalized score
fuzzer
libafl_fuzzbench_value_profile 80.70
libafl_fuzzbench_weighted 79.40
libafl_fuzzbench_rand_scheduler 78.26
aflplusplus 76.26
libfuzzer 17.42
honggfuzz 11.76
afl 10.47
centipede 0.32
aflfast 0.00
aflsmart 0.00
eclipser 0.00
fairfuzz 0.00
mopt 0.00
By avg. rank
average rank
fuzzer
aflplusplus 1.76
libafl_fuzzbench_value_profile 2.29
libafl_fuzzbench_weighted 2.71
libafl_fuzzbench_rand_scheduler 3.24
honggfuzz 4.24
libfuzzer 4.41
afl 4.59
centipede 4.65
aflfast 4.71
aflsmart 4.71
eclipser 4.71
fairfuzz 4.71
mopt 4.71
  • Critical difference diagram
    The diagram visualizes the average rank of fuzzers (second ranking above) while showing the significance of the differences as well. What is considered a "critical difference" (CD) is based on the Friedman/Nemenyi post-hoc test. See more in the documentation.
    Note: If a fuzzer does not support all benchmarks, its ranking as shown in this diagram can be lower than it should be. So please check the list of supported benchmarks for the fuzzer(s) of your interest. The list could be specified in the fuzzer's README.md like this.
  • Median relative code-coverages on each benchmark

    Note: The relative coverage summary table shows the median relative performance of each fuzzer to the experiment maximum. Thus the highest relative performance may not be 100%.
    trial_relative_coverage = trial_coverage / experiment_max_coverage

      libafl_fuzzbench_value_profile aflplusplus libafl_fuzzbench_weighted libafl_fuzzbench_rand_scheduler libfuzzer honggfuzz afl centipede aflfast aflsmart eclipser fairfuzz mopt
    FuzzerMedian 98.00 99.00 99.00 97.00 95.00 48.00 47.00 5.00 0.00 0.00 0.00 0.00 0.00
    FuzzerMean 87.60 87.57 86.33 85.33 80.67 48.67 41.67 5.00 0.00 0.00 0.00 0.00 0.00
    arduinojson_json_fuzzer 99.00 100.00 99.00 99.00 nan nan nan nan nan nan nan nan nan
    assimp_assimp_fuzzer 61.00 nan 57.00 51.00 nan nan nan nan nan nan nan nan nan
    astc-encoder_fuzz_astc_physical_to_symbolic 98.00 nan 95.00 95.00 nan nan nan nan nan nan nan nan nan
    botan_tls_server nan 48.00 nan nan 48.00 48.00 47.00 nan nan nan nan nan nan
    brotli_decode_fuzzer 99.00 nan 99.00 99.00 nan nan nan nan nan nan nan nan nan
    double-conversion_string_to_double_fuzzer 98.00 99.00 97.00 97.00 nan nan nan nan nan nan nan nan nan
    draco_draco_pc_decoder_fuzzer 78.00 92.00 65.00 62.00 nan nan nan nan nan nan nan nan nan
    dropbear_fuzzer-postauth_nomaths nan 96.00 nan nan 95.00 98.00 78.00 nan nan nan nan nan nan
    firestore_firestore_serializer_fuzzer 100.00 100.00 100.00 100.00 nan nan nan nan nan nan nan nan nan
    fmt_chrono-duration-fuzzer 98.00 99.00 99.00 99.00 nan nan nan nan nan nan nan nan nan
    guetzli_guetzli_fuzzer 99.00 99.00 99.00 97.00 nan nan nan nan nan nan nan nan nan
    icu_unicode_string_codepage_create_fuzzer 99.00 99.00 99.00 99.00 nan nan nan nan nan nan nan nan nan
    jansson_json_load_dump_fuzzer 99.00 99.00 99.00 99.00 nan nan nan nan nan nan nan nan nan
    libaom_av1_dec_fuzzer 96.00 97.00 98.00 94.00 nan nan nan nan nan nan nan nan nan
    libcoap_pdu_parse_fuzzer 91.00 99.00 90.00 90.00 nan nan nan nan nan nan nan nan nan
    libhevc_hevc_dec_fuzzer 99.00 99.00 99.00 99.00 nan nan nan 5.00 0.00 0.00 0.00 0.00 0.00
    librdkafka_fuzz_regex 0.00 0.00 0.00 0.00 99.00 0.00 0.00 nan nan nan nan nan nan
    • Fuzzers are sorted by "FuzzerMean" (average median relative coverage), highest on the left.
    • Green background = highest relative median coverage.
    • Blue gradient background = greater than 95% relative median coverage.

arduinojson_json_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus 82800 20.0 401.600000 0.502625 401.0 401.0 402.0 402.00 402.0
    libafl_fuzzbench_value_profile 82800 19.0 400.368421 0.597265 399.0 400.0 400.0 401.00 401.0
    libafl_fuzzbench_rand_scheduler 82800 19.0 398.421053 0.961237 397.0 398.0 398.0 399.00 400.0
    libafl_fuzzbench_weighted 82800 18.0 398.722222 1.074055 397.0 398.0 398.0 399.75 401.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

assimp_assimp_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
error
The following fuzzers do not have enough samples: libafl_fuzzbench_value_profile.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libafl_fuzzbench_value_profile 5400 14.0 2633.000000 277.346493 2090.0 2445.75 2657.0 2846.25 3017.0
    libafl_fuzzbench_weighted 5400 19.0 2515.789474 241.512268 2027.0 2392.50 2520.0 2729.50 2809.0
    libafl_fuzzbench_rand_scheduler 5400 17.0 2286.117647 136.140039 2103.0 2200.00 2239.0 2396.00 2585.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

astc-encoder_fuzz_astc_physical_to_symbolic summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libafl_fuzzbench_value_profile 82800 17.0 503.235294 1.888432 500.0 502.0 503.0 504.0 507.0
    libafl_fuzzbench_weighted 82800 17.0 490.529412 2.960525 488.0 489.0 489.0 492.0 498.0
    libafl_fuzzbench_rand_scheduler 82800 18.0 488.777778 1.003263 488.0 488.0 488.5 489.0 491.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

botan_tls_server summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
error
The following fuzzers do not have enough samples: afl, honggfuzz, libfuzzer.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    honggfuzz 3600 10.0 1365.1 188.481328 1171.0 1242.75 1252.5 1549.00 1671.0
    aflplusplus 3600 20.0 1333.2 203.259906 1179.0 1224.00 1252.0 1288.75 1792.0
    libfuzzer 3600 10.0 1217.0 34.062035 1176.0 1178.25 1241.5 1242.00 1248.0
    afl 3600 10.0 1239.9 6.172520 1235.0 1236.25 1237.5 1241.25 1255.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

brotli_decode_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libafl_fuzzbench_weighted 82800 17.0 902.470588 1.504894 900.0 901.0 902.0 904.0 904.0
    libafl_fuzzbench_rand_scheduler 82800 17.0 898.705882 3.235829 894.0 895.0 900.0 901.0 904.0
    libafl_fuzzbench_value_profile 82800 17.0 898.000000 3.905125 893.0 894.0 899.0 901.0 903.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

double-conversion_string_to_double_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus 82800 20.0 507.750000 1.831738 504.0 506.00 508.0 509.0 510.0
    libafl_fuzzbench_value_profile 82800 18.0 501.388889 1.243283 500.0 500.25 501.0 502.0 504.0
    libafl_fuzzbench_rand_scheduler 82800 18.0 496.833333 2.093407 491.0 496.00 497.0 498.0 501.0
    libafl_fuzzbench_weighted 82800 18.0 496.833333 2.617812 491.0 495.25 496.5 498.0 502.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

draco_draco_pc_decoder_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
error
The following fuzzers do not have enough samples: libafl_fuzzbench_value_profile.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus 82800 15.0 1850.066667 96.068775 1698.0 1779.00 1839.0 1931.50 1996.0
    libafl_fuzzbench_value_profile 82800 14.0 1544.642857 137.732521 1270.0 1482.00 1563.5 1618.25 1739.0
    libafl_fuzzbench_weighted 82800 18.0 1326.277778 99.544732 1066.0 1291.75 1316.5 1369.25 1502.0
    libafl_fuzzbench_rand_scheduler 82800 18.0 1288.888889 170.778456 1024.0 1155.50 1250.5 1426.75 1584.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

dropbear_fuzzer-postauth_nomaths summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    honggfuzz 3600 10.0 883.6 9.570789 860.0 881.25 885.5 889.0 895.0
    aflplusplus 3600 10.0 863.0 2.538591 858.0 861.00 864.5 865.0 865.0
    libfuzzer 3600 10.0 857.1 5.877452 841.0 857.00 858.5 860.0 861.0
    afl 3600 10.0 701.0 0.000000 701.0 701.00 701.0 701.0 701.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

firestore_firestore_serializer_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus 82800 20.0 287.000000 0.000000 287.0 287.0 287.0 287.0 287.0
    libafl_fuzzbench_rand_scheduler 82800 19.0 286.263158 0.991189 285.0 285.0 287.0 287.0 287.0
    libafl_fuzzbench_value_profile 82800 18.0 287.000000 0.000000 287.0 287.0 287.0 287.0 287.0
    libafl_fuzzbench_weighted 82800 17.0 286.294118 0.985184 285.0 285.0 287.0 287.0 287.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

fmt_chrono-duration-fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus 82800 20.0 1089.800000 3.607011 1081.0 1086.75 1091.0 1093.00 1095.0
    libafl_fuzzbench_rand_scheduler 82800 18.0 1089.888889 3.178769 1083.0 1088.00 1090.0 1091.75 1097.0
    libafl_fuzzbench_weighted 82800 17.0 1086.823529 3.778811 1081.0 1084.00 1087.0 1091.00 1092.0
    libafl_fuzzbench_value_profile 82800 19.0 1085.157895 3.354320 1080.0 1083.50 1085.0 1086.00 1093.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

guetzli_guetzli_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus 82800 20.0 1498.300000 5.391026 1484.0 1495.0 1498.5 1502.25 1506.0
    libafl_fuzzbench_value_profile 82800 16.0 1490.562500 7.164438 1474.0 1492.0 1493.0 1493.25 1500.0
    libafl_fuzzbench_weighted 82800 18.0 1492.000000 6.019576 1481.0 1488.0 1493.0 1497.75 1500.0
    libafl_fuzzbench_rand_scheduler 82800 19.0 1479.210526 9.823888 1465.0 1472.0 1474.0 1490.00 1494.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

icu_unicode_string_codepage_create_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
error
The following fuzzers do not have enough samples: libafl_fuzzbench_value_profile.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus 82800 20.0 1339.850000 0.745160 1339.0 1339.0 1340.0 1340.00 1341.0
    libafl_fuzzbench_value_profile 82800 14.0 1329.285714 18.243771 1301.0 1311.5 1340.0 1340.75 1341.0
    libafl_fuzzbench_weighted 82800 18.0 1338.666667 0.840168 1337.0 1338.0 1339.0 1339.00 1340.0
    libafl_fuzzbench_rand_scheduler 82800 19.0 1336.578947 8.927584 1300.0 1338.0 1338.0 1339.00 1341.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

jansson_json_load_dump_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libafl_fuzzbench_value_profile 82800 18.0 770.777778 0.942809 769.0 770.00 771.0 771.75 772.0
    aflplusplus 82800 20.0 767.800000 1.472556 765.0 767.00 768.0 769.00 770.0
    libafl_fuzzbench_rand_scheduler 82800 18.0 767.166667 1.465285 764.0 766.25 767.5 768.00 769.0
    libafl_fuzzbench_weighted 82800 20.0 766.900000 1.293709 765.0 766.00 767.0 768.00 770.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

libaom_av1_dec_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libafl_fuzzbench_weighted 82800 20.0 11214.150000 119.779304 11039.0 11111.00 11197.5 11315.25 11388.0
    aflplusplus 82800 20.0 11048.450000 295.341457 10282.0 10997.75 11104.5 11242.25 11352.0
    libafl_fuzzbench_value_profile 82800 18.0 11021.055556 133.615107 10699.0 10954.75 11012.0 11065.75 11285.0
    libafl_fuzzbench_rand_scheduler 82800 16.0 10752.625000 138.347329 10512.0 10630.50 10767.5 10892.50 10922.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

libcoap_pdu_parse_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus 82800 20.0 817.350000 1.268028 815.0 817.00 817.0 818.0 821.0
    libafl_fuzzbench_value_profile 82800 17.0 751.117647 0.696631 750.0 751.00 751.0 752.0 752.0
    libafl_fuzzbench_weighted 82800 18.0 745.166667 1.504894 742.0 744.25 745.5 746.0 748.0
    libafl_fuzzbench_rand_scheduler 82800 19.0 744.789474 1.182227 743.0 744.00 745.0 745.5 747.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

libhevc_hevc_dec_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
error
The following fuzzers do not have enough samples: libafl_fuzzbench_weighted.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus 82800 20.0 10332.700000 47.309841 10219.0 10331.5 10355.0 10360.0 10375.0
    libafl_fuzzbench_value_profile 82800 17.0 10329.705882 8.289788 10307.0 10325.0 10330.0 10333.0 10343.0
    libafl_fuzzbench_weighted 82800 15.0 10321.133333 10.425701 10304.0 10315.0 10321.0 10326.5 10339.0
    libafl_fuzzbench_rand_scheduler 82800 17.0 10303.117647 16.419814 10272.0 10293.0 10306.0 10315.0 10335.0
    centipede 82800 20.0 1260.300000 1723.585892 555.0 555.0 557.0 557.0 5651.0
    aflfast 82800 20.0 0.000000 0.000000 0.0 0.0 0.0 0.0 0.0
    aflsmart 82800 20.0 0.000000 0.000000 0.0 0.0 0.0 0.0 0.0
    eclipser 82800 17.0 0.000000 0.000000 0.0 0.0 0.0 0.0 0.0
    fairfuzz 82800 20.0 0.000000 0.000000 0.0 0.0 0.0 0.0 0.0
    mopt 82800 20.0 0.000000 0.000000 0.0 0.0 0.0 0.0 0.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

librdkafka_fuzz_regex summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
error
The following fuzzers do not have enough samples: afl, aflplusplus, honggfuzz, libfuzzer.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libfuzzer 900 10.0 379.8 4.661902 367.0 381.0 381.5 382.0 382.0
    afl 900 10.0 148.0 191.110788 0.0 0.0 0.0 367.0 378.0
    aflplusplus 900 10.0 73.4 154.752275 0.0 0.0 0.0 0.0 371.0
    honggfuzz 900 10.0 68.1 143.576886 0.0 0.0 0.0 0.0 344.0
    libafl_fuzzbench_rand_scheduler 900 18.0 0.0 0.000000 0.0 0.0 0.0 0.0 0.0
    libafl_fuzzbench_value_profile 900 20.0 19.0 84.970583 0.0 0.0 0.0 0.0 380.0
    libafl_fuzzbench_weighted 900 20.0 0.0 0.000000 0.0 0.0 0.0 0.0 0.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

experiment data

You can download the raw data for this report here.

Check out the documentation on how to create customized reports using this data. Also see some example Colab notebooks for doing custom analysis on the data here.

Experiment Description:

(None,)