FuzzBench: 2024-08-13-aflchurnplusplus report

experiment summary

No aggregate ranking as the data contains a single benchmark.
  • Median relative code-coverages on each benchmark

    Note: The relative coverage summary table shows the median relative performance of each fuzzer to the experiment maximum. Thus the highest relative performance may not be 100%.
    trial_relative_coverage = trial_coverage / experiment_max_coverage

      libafl afl aflchurnplusplus aflplusplus libfuzzer
    FuzzerMedian 98.00 96.00 95.00 95.00 95.00
    FuzzerMean 98.00 96.00 95.00 95.00 95.00
    arrow_arrow-ipc-stream-fuzz_1a34a0 98.00 96.00 95.00 95.00 95.00
    • Fuzzers are sorted by "FuzzerMean" (average median relative coverage), highest on the left.
    • Green background = highest relative median coverage.
    • Blue gradient background = greater than 95% relative median coverage.
  • Median relative bug-coverages on each benchmark

    Note: The relative coverage summary table shows the median relative performance of each fuzzer to the experiment maximum. Thus the highest relative performance may not be 100%.
    trial_relative_coverage = trial_coverage / experiment_max_coverage

      afl aflchurnplusplus aflplusplus libafl libfuzzer
    FuzzerMedian 0.00 0.00 0.00 0.00 0.00
    FuzzerMean 0.00 0.00 0.00 0.00 0.00
    arrow_arrow-ipc-stream-fuzz_1a34a0 0.00 0.00 0.00 0.00 0.00
    • Fuzzers are sorted by "FuzzerMean" (average median relative coverage), highest on the left.
    • Green background = highest relative median coverage.
    • Blue gradient background = greater than 95% relative median coverage.
  • Total unique bugs found on each benchmark
      afl aflchurnplusplus aflplusplus libafl libfuzzer Total
    FuzzerSum 0 0 0 0 0 0
    arrow_arrow-ipc-stream-fuzz_1a34a0 0 0 0 0 0 0
    • Fuzzers are sorted by "FuzzerSum", highest on the left.
    • Green background = most unique bugs found.
    • *note: This table represents unique bugs found across all trials.

arrow_arrow-ipc-stream-fuzz_1a34a0 summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
error
The following fuzzers do not have enough samples: aflchurnplusplus.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    aflchurnplusplus 82800 15.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    aflplusplus 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    libafl 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    libfuzzer 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libafl 82800 20.0 2439.85 54.841662 2336.0 2387.50 2459.0 2489.50 2501.0
    afl 82800 20.0 2399.10 45.588433 2302.0 2365.25 2419.0 2430.00 2455.0
    aflplusplus 82800 20.0 2375.40 50.568765 2304.0 2320.25 2395.5 2415.25 2449.0
    libfuzzer 82800 20.0 2388.75 36.693431 2314.0 2374.50 2394.0 2413.25 2451.0
    aflchurnplusplus 82800 15.0 2388.00 57.583480 2302.0 2337.00 2388.0 2439.50 2459.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

experiment data

You can download the raw data for this report here.

Check out the documentation on how to create customized reports using this data. Also see some example Colab notebooks for doing custom analysis on the data here.

Experiment Description:

(None,)