FuzzBench: 2024-08-18-aflchurnplusplus report

experiment summary

We show two different aggregate (cross-benchmark) rankings of fuzzers. The first is based on the average of per-benchmarks scores, where the score represents the percentage of the highest reached median bug-coverage on a given benchmark (higher value is better). The second ranking shows the average rank of fuzzers, after we rank them on each benchmark according to their median reached bug-covereges (lower value is better).
By avg. score
average normalized score
fuzzer
aflchurnplusplus 93.33
afl 86.67
aflplusplus 66.67
By avg. rank
average rank
fuzzer
aflchurnplusplus 1.17
afl 1.33
aflplusplus 1.67
  • Critical difference diagram
    The diagram visualizes the average rank of fuzzers (second ranking above) while showing the significance of the differences as well. What is considered a "critical difference" (CD) is based on the Friedman/Nemenyi post-hoc test. See more in the documentation.
    Note: If a fuzzer does not support all benchmarks, its ranking as shown in this diagram can be lower than it should be. So please check the list of supported benchmarks for the fuzzer(s) of your interest. The list could be specified in the fuzzer's README.md like this.
  • Median relative code-coverages on each benchmark

    Note: The relative coverage summary table shows the median relative performance of each fuzzer to the experiment maximum. Thus the highest relative performance may not be 100%.
    trial_relative_coverage = trial_coverage / experiment_max_coverage

      aflchurnplusplus aflplusplus afl
    FuzzerMedian 94.50 94.50 93.00
    FuzzerMean 94.67 91.33 84.83
    arrow_arrow-ipc-stream-fuzz_eee13b 99.00 98.00 98.00
    bloaty_fuzz_target_f01ea5 96.00 95.00 95.00
    ffmpeg_ffmpeg_demuxer_fuzzer_fe85af 92.00 90.00 66.00
    grok_grk_decompress_fuzzer_d9ff920 95.00 95.00 93.00
    lcms_cms_transform_all_fuzzer_a9796f 92.00 76.00 64.00
    libpcap_fuzz_filter_bc594f 94.00 94.00 93.00
    • Fuzzers are sorted by "FuzzerMean" (average median relative coverage), highest on the left.
    • Green background = highest relative median coverage.
    • Blue gradient background = greater than 95% relative median coverage.
  • Median relative bug-coverages on each benchmark

    Note: The relative coverage summary table shows the median relative performance of each fuzzer to the experiment maximum. Thus the highest relative performance may not be 100%.
    trial_relative_coverage = trial_coverage / experiment_max_coverage

      aflchurnplusplus afl aflplusplus
    FuzzerMedian 58.00 41.50 50.00
    FuzzerMean 54.00 51.33 44.33
    arrow_arrow-ipc-stream-fuzz_eee13b 66.00 100.00 66.00
    bloaty_fuzz_target_f01ea5 100.00 100.00 100.00
    ffmpeg_ffmpeg_demuxer_fuzzer_fe85af 75.00 25.00 50.00
    grok_grk_decompress_fuzzer_d9ff920 50.00 50.00 50.00
    lcms_cms_transform_all_fuzzer_a9796f 33.00 33.00 0.00
    libpcap_fuzz_filter_bc594f 0.00 0.00 0.00
    • Fuzzers are sorted by "FuzzerMean" (average median relative coverage), highest on the left.
    • Green background = highest relative median coverage.
    • Blue gradient background = greater than 95% relative median coverage.
  • Total unique bugs found on each benchmark
      Total aflplusplus aflchurnplusplus afl
    FuzzerSum 27 24 23 17
    arrow_arrow-ipc-stream-fuzz_eee13b 5 5 5 5
    bloaty_fuzz_target_f01ea5 1 1 1 1
    ffmpeg_ffmpeg_demuxer_fuzzer_fe85af 13 11 11 7
    grok_grk_decompress_fuzzer_d9ff920 3 3 2 2
    lcms_cms_transform_all_fuzzer_a9796f 5 4 4 2
    libpcap_fuzz_filter_bc594f 0 0 0 0
    • Fuzzers are sorted by "FuzzerSum", highest on the left.
    • Green background = most unique bugs found.
    • *note: This table represents unique bugs found across all trials.

arrow_arrow-ipc-stream-fuzz_eee13b summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 2.9 0.307794 2.0 3.0 3.0 3.0 3.0
    aflchurnplusplus 82800 20.0 2.2 0.410391 2.0 2.0 2.0 2.0 3.0
    aflplusplus 82800 20.0 2.3 0.470162 2.0 2.0 2.0 3.0 3.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflchurnplusplus 82800 20.0 2341.20 46.771111 2235.0 2342.25 2358.5 2364.00 2382.0
    aflplusplus 82800 20.0 2339.90 45.872707 2227.0 2335.25 2357.5 2368.75 2378.0
    afl 82800 20.0 2336.35 26.752816 2239.0 2333.00 2338.5 2352.00 2366.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

bloaty_fuzz_target_f01ea5 summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 1.0 0.000000 1.0 1.0 1.0 1.0 1.0
    aflchurnplusplus 82800 20.0 0.8 0.410391 0.0 1.0 1.0 1.0 1.0
    aflplusplus 82800 20.0 1.0 0.000000 1.0 1.0 1.0 1.0 1.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflchurnplusplus 82800 20.0 5966.10 102.149478 5759.0 5900.00 5942.5 6051.00 6173.0
    aflplusplus 82800 20.0 5918.85 88.106559 5766.0 5855.50 5908.5 5955.50 6143.0
    afl 82800 20.0 5880.80 124.185600 5666.0 5780.25 5879.0 5984.25 6057.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

ffmpeg_ffmpeg_demuxer_fuzzer_fe85af summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflchurnplusplus 82800 20.0 2.60 0.598243 2.0 2.0 3.0 3.0 4.0
    aflplusplus 82800 20.0 2.35 0.489360 2.0 2.0 2.0 3.0 3.0
    afl 82800 20.0 1.55 0.686333 1.0 1.0 1.0 2.0 3.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflchurnplusplus 82800 20.0 19962.80 887.545438 17539.0 19596.00 20049.5 20462.50 21599.0
    aflplusplus 82800 20.0 19484.55 700.475739 18253.0 19041.25 19442.0 19861.25 20809.0
    afl 82800 20.0 14248.65 454.046285 13226.0 13908.50 14370.5 14567.75 15067.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

grok_grk_decompress_fuzzer_d9ff920 summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 1.00 0.000000 1.0 1.0 1.0 1.0 1.0
    aflchurnplusplus 82800 20.0 1.00 0.000000 1.0 1.0 1.0 1.0 1.0
    aflplusplus 82800 20.0 1.05 0.223607 1.0 1.0 1.0 1.0 2.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus 82800 20.0 5826.6 308.476119 4925.0 5834.75 5902.0 5937.25 6163.0
    aflchurnplusplus 82800 20.0 5837.1 128.827057 5314.0 5834.75 5861.5 5895.00 5933.0
    afl 82800 20.0 5737.8 128.739885 5291.0 5708.25 5745.0 5804.75 5901.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

lcms_cms_transform_all_fuzzer_a9796f summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 0.850000 0.875094 0.0 0.0 1.0 2.0 2.0
    aflchurnplusplus 82800 17.0 1.352941 0.606339 0.0 1.0 1.0 2.0 2.0
    aflplusplus 82800 20.0 0.300000 0.732695 0.0 0.0 0.0 0.0 3.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflchurnplusplus 82800 17.0 2057.823529 253.772840 1391.0 1950.00 2162.0 2198.00 2338.0
    aflplusplus 82800 20.0 1812.400000 148.925061 1604.0 1680.50 1785.0 1929.00 2098.0
    afl 82800 20.0 1522.000000 161.702781 1270.0 1422.25 1516.5 1577.25 1861.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

libpcap_fuzz_filter_bc594f summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    aflchurnplusplus 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    aflplusplus 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflchurnplusplus 82800 20.0 3399.50 77.870000 3293.0 3356.50 3377.0 3435.25 3556.0
    aflplusplus 82800 20.0 3368.20 71.919985 3278.0 3309.75 3359.5 3406.00 3499.0
    afl 82800 20.0 3339.25 42.054945 3247.0 3315.75 3341.0 3368.50 3415.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

experiment data

You can download the raw data for this report here.

Check out the documentation on how to create customized reports using this data. Also see some example Colab notebooks for doing custom analysis on the data here.

Experiment Description:

(None,)