FuzzBench: 2024-05-17-new-bug report

experiment summary

We show two different aggregate (cross-benchmark) rankings of fuzzers. The first is based on the average of per-benchmarks scores, where the score represents the percentage of the highest reached median bug-coverage on a given benchmark (higher value is better). The second ranking shows the average rank of fuzzers, after we rank them on each benchmark according to their median reached bug-covereges (lower value is better).
By avg. score
average normalized score
fuzzer
afl 100.00
honggfuzz 100.00
libfuzzer 100.00
aflplusplus 66.67
libafl 33.33
By avg. rank
average rank
fuzzer
afl 1.0
honggfuzz 1.0
libfuzzer 1.0
aflplusplus 2.0
libafl 3.0
  • Critical difference diagram
    The diagram visualizes the average rank of fuzzers (second ranking above) while showing the significance of the differences as well. What is considered a "critical difference" (CD) is based on the Friedman/Nemenyi post-hoc test. See more in the documentation.
    Note: If a fuzzer does not support all benchmarks, its ranking as shown in this diagram can be lower than it should be. So please check the list of supported benchmarks for the fuzzer(s) of your interest. The list could be specified in the fuzzer's README.md like this.
  • Median relative code-coverages on each benchmark

    Note: The relative coverage summary table shows the median relative performance of each fuzzer to the experiment maximum. Thus the highest relative performance may not be 100%.
    trial_relative_coverage = trial_coverage / experiment_max_coverage

      afl aflplusplus honggfuzz libfuzzer libafl
    FuzzerMedian 97.00 96.00 96.00 93.00 91.50
    FuzzerMean 97.25 95.50 93.75 92.75 91.50
    bloaty_fuzz_target_52948c 97.00 92.00 94.00 90.00 84.00
    harfbuzz_hb-shape-fuzzer_17863b 98.00 97.00 98.00 96.00 nan
    libxml2_xml_e85b9b 97.00 95.00 84.00 88.00 99.00
    php_php-fuzz-parser_0dbedb 97.00 98.00 99.00 97.00 nan
    • Fuzzers are sorted by "FuzzerMean" (average median relative coverage), highest on the left.
    • Green background = highest relative median coverage.
    • Blue gradient background = greater than 95% relative median coverage.
  • Median relative bug-coverages on each benchmark

    Note: The relative coverage summary table shows the median relative performance of each fuzzer to the experiment maximum. Thus the highest relative performance may not be 100%.
    trial_relative_coverage = trial_coverage / experiment_max_coverage

      afl honggfuzz libfuzzer aflplusplus libafl
    FuzzerMedian 100.00 100.00 100.00 50.00 50.00
    FuzzerMean 75.00 75.00 75.00 50.00 50.00
    bloaty_fuzz_target_52948c 100.00 100.00 100.00 100.00 0.00
    harfbuzz_hb-shape-fuzzer_17863b 100.00 100.00 100.00 100.00 nan
    libxml2_xml_e85b9b 100.00 100.00 100.00 0.00 100.00
    php_php-fuzz-parser_0dbedb 0.00 0.00 0.00 0.00 nan
    • Fuzzers are sorted by "FuzzerMean" (average median relative coverage), highest on the left.
    • Green background = highest relative median coverage.
    • Blue gradient background = greater than 95% relative median coverage.
  • Total unique bugs found on each benchmark
      Total afl libfuzzer honggfuzz aflplusplus libafl
    FuzzerSum 11 10 10 8 7 3
    bloaty_fuzz_target_52948c 1 1 1 1 1 1
    harfbuzz_hb-shape-fuzzer_17863b 7 6 6 4 3 nan
    libxml2_xml_e85b9b 2 2 2 2 2 2
    php_php-fuzz-parser_0dbedb 1 1 1 1 1 nan
    • Fuzzers are sorted by "FuzzerSum", highest on the left.
    • Green background = most unique bugs found.
    • *note: This table represents unique bugs found across all trials.

bloaty_fuzz_target_52948c summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
error
The following fuzzers do not have enough samples: aflplusplus.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 1.000000 0.000000 1.0 1.0 1.0 1.0 1.0
    aflplusplus 82800 15.0 0.866667 0.351866 0.0 1.0 1.0 1.0 1.0
    honggfuzz 82800 20.0 0.650000 0.489360 0.0 0.0 1.0 1.0 1.0
    libfuzzer 82800 20.0 0.950000 0.223607 0.0 1.0 1.0 1.0 1.0
    libafl 82800 19.0 0.315789 0.477567 0.0 0.0 0.0 1.0 1.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 5952.700000 123.164033 5656.0 5896.50 5987.5 6028.00 6134.0
    honggfuzz 82800 20.0 5767.350000 81.342858 5620.0 5686.25 5791.0 5812.25 5962.0
    aflplusplus 82800 15.0 5640.466667 167.770620 5372.0 5514.50 5679.0 5782.50 5871.0
    libfuzzer 82800 20.0 5538.250000 134.959984 5336.0 5437.75 5558.5 5630.50 5764.0
    libafl 82800 19.0 5187.000000 287.319605 4686.0 5049.50 5206.0 5295.00 6001.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

harfbuzz_hb-shape-fuzzer_17863b summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 1.0 0.0 1.0 1.0 1.0 1.0 1.0
    aflplusplus 82800 20.0 1.0 0.0 1.0 1.0 1.0 1.0 1.0
    honggfuzz 82800 20.0 1.0 0.0 1.0 1.0 1.0 1.0 1.0
    libfuzzer 82800 20.0 1.0 0.0 1.0 1.0 1.0 1.0 1.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 10140.4 53.221879 10048.0 10114.25 10138.0 10168.00 10268.0
    honggfuzz 82800 20.0 10136.0 36.743779 10063.0 10114.50 10130.5 10165.00 10191.0
    aflplusplus 82800 20.0 10002.8 103.029173 9728.0 9996.50 10031.5 10050.75 10140.0
    libfuzzer 82800 20.0 9910.1 69.981125 9732.0 9883.25 9919.0 9935.25 10050.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

libxml2_xml_e85b9b summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 0.95 0.223607 0.0 1.0 1.0 1.0 1.0
    honggfuzz 82800 20.0 0.55 0.510418 0.0 0.0 1.0 1.0 1.0
    libafl 82800 20.0 1.00 0.000000 1.0 1.0 1.0 1.0 1.0
    libfuzzer 82800 20.0 0.55 0.510418 0.0 0.0 1.0 1.0 1.0
    aflplusplus 82800 20.0 0.15 0.366348 0.0 0.0 0.0 0.0 1.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libafl 82800 20.0 19859.15 128.118479 19649.0 19744.25 19880.0 19927.00 20070.0
    afl 82800 20.0 19658.80 179.473675 19302.0 19613.50 19650.0 19771.50 20005.0
    aflplusplus 82800 20.0 18912.10 857.017282 16430.0 19038.50 19188.0 19284.25 19443.0
    libfuzzer 82800 20.0 17645.80 1000.839574 16318.0 16460.75 17750.5 18674.50 18905.0
    honggfuzz 82800 20.0 17020.35 90.315398 16792.0 16999.25 17035.5 17081.25 17133.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

php_php-fuzz-parser_0dbedb summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 0.20 0.410391 0.0 0.0 0.0 0.0 1.0
    aflplusplus 82800 20.0 0.10 0.307794 0.0 0.0 0.0 0.0 1.0
    honggfuzz 82800 20.0 0.10 0.307794 0.0 0.0 0.0 0.0 1.0
    libfuzzer 82800 20.0 0.15 0.366348 0.0 0.0 0.0 0.0 1.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    honggfuzz 82800 20.0 17011.75 114.677523 16764.0 16944.5 17051.0 17110.25 17141.0
    aflplusplus 82800 20.0 16869.25 116.340641 16715.0 16770.5 16830.0 16959.25 17056.0
    libfuzzer 82800 20.0 16757.55 45.219319 16689.0 16731.5 16750.5 16770.25 16904.0
    afl 82800 20.0 16642.80 46.548899 16566.0 16618.5 16639.5 16658.50 16804.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

experiment data

You can download the raw data for this report here.

Check out the documentation on how to create customized reports using this data. Also see some example Colab notebooks for doing custom analysis on the data here.

Experiment Description:

(None,)