FuzzBench: differential-plots-3 report

warning
Please consider this as a preliminary report to demonstrate the capabilities of FuzzBench. While we have tried our best, we have not confirmed that we configured everything correctly. We are hoping to work together with the community to validate results and improve the set of fuzzers, benchmarks, and their configurations in the future. See FAQ for more details.

experiment summary

We show two different aggregate (cross-benchmark) rankings of fuzzers. The first is based on the average of per-benchmarks scores, where the score represents the percentage of the highest reached median coverage on a given benchmark (higher value is better). The second ranking shows the average rank of fuzzers, after we rank them on each benchmark according to their median reached covereges (lower value is better).
By avg. score
average normalized score
fuzzer
afl 98.62
honggfuzz 97.28
lafintel 95.49
libfuzzer 94.47
mopt 83.85
By avg. rank
average rank
fuzzer
honggfuzz 2.00
afl 2.33
mopt 3.00
lafintel 3.33
libfuzzer 4.33
  • Critical difference diagram
    The diagram visualizes the average rank of fuzzers (second ranking above) while showing the significance of the differences as well. What is considered a "critical difference" (CD) is based on the Friedman/Nemenyi post-hoc test. See more in the documentation.
    Note: If a fuzzer does not support all benchmarks, its ranking as shown in this diagram can be lower than it should be. So please check the list of supported benchmarks for the fuzzer(s) of your interest. The list could be specified in the fuzzer's README.md like this.
  • Median coverages on each benchmark
    fuzzer afl honggfuzz lafintel libfuzzer mopt
    benchmark
    bloaty_fuzz_target 7132.5 6575.5 6395.0 6478.5 7157.5
    libpng-1.2.56 1940.0 2016.5 1959.5 1877.5 1940.5
    openssl_x509 13721.0 13719.0 13713.5 13691.5 7592.0

bloaty_fuzz_target summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    mopt 1800 2.0 7157.5 45.961941 7125.0 7141.25 7157.5 7173.75 7190.0
    afl 1800 2.0 7132.5 23.334524 7116.0 7124.25 7132.5 7140.75 7149.0
    honggfuzz 1800 2.0 6575.5 79.903066 6519.0 6547.25 6575.5 6603.75 6632.0
    libfuzzer 1800 2.0 6478.5 166.170094 6361.0 6419.75 6478.5 6537.25 6596.0
    lafintel 1800 2.0 6395.0 70.710678 6345.0 6370.00 6395.0 6420.00 6445.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique coverage plots
    Ranking by unique edges covered
    Each bar shows the total number of edges found by a given fuzzer. The colored area shows the number of unique edges (i.e., edges that were not covered by any other fuzzers).
    Pairwise unique coverage
    Each cell represents the number of edges covered by the fuzzer of the column but not by the fuzzer of the row

libpng-1.2.56 summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    honggfuzz 1800 2.0 2016.5 24.748737 1999.0 2007.75 2016.5 2025.25 2034.0
    lafintel 1800 2.0 1959.5 0.707107 1959.0 1959.25 1959.5 1959.75 1960.0
    mopt 1800 2.0 1940.5 2.121320 1939.0 1939.75 1940.5 1941.25 1942.0
    afl 1800 2.0 1940.0 1.414214 1939.0 1939.50 1940.0 1940.50 1941.0
    libfuzzer 1800 2.0 1877.5 50.204581 1842.0 1859.75 1877.5 1895.25 1913.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique coverage plots
    Ranking by unique edges covered
    Each bar shows the total number of edges found by a given fuzzer. The colored area shows the number of unique edges (i.e., edges that were not covered by any other fuzzers).
    Pairwise unique coverage
    Each cell represents the number of edges covered by the fuzzer of the column but not by the fuzzer of the row

openssl_x509 summary

Ranking by median reached coverage
Reached coverage distribution
Mean coverage growth over time
* The error bands show the 95% confidence interval around the mean coverage.
  • Sample statistics and statistical significance
    Coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 1800 2.0 13721.0 9.899495 13714.0 13717.50 13721.0 13724.50 13728.0
    honggfuzz 1800 2.0 13719.0 8.485281 13713.0 13716.00 13719.0 13722.00 13725.0
    lafintel 1800 2.0 13713.5 0.707107 13713.0 13713.25 13713.5 13713.75 13714.0
    libfuzzer 1800 2.0 13691.5 2.121320 13690.0 13690.75 13691.5 13692.25 13693.0
    mopt 1800 2.0 7592.0 8677.614419 1456.0 4524.00 7592.0 10660.00 13728.0

    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique coverage plots
    Ranking by unique edges covered
    Each bar shows the total number of edges found by a given fuzzer. The colored area shows the number of unique edges (i.e., edges that were not covered by any other fuzzers).
    Pairwise unique coverage
    Each cell represents the number of edges covered by the fuzzer of the column but not by the fuzzer of the row

experiment data

You can download the raw data for this report here.

Check out the documentation on how to create customized reports using this data. Also see some example Colab notebooks for doing custom analysis on the data here.

The experiment was conducted using this FuzzBench commit: 79a86a5d247587c87aad8c715ff5c673c8d9207e