FuzzBench: 2024-08-14-aflchurnplusplus report

experiment summary

We show two different aggregate (cross-benchmark) rankings of fuzzers. The first is based on the average of per-benchmarks scores, where the score represents the percentage of the highest reached median bug-coverage on a given benchmark (higher value is better). The second ranking shows the average rank of fuzzers, after we rank them on each benchmark according to their median reached bug-covereges (lower value is better).
By avg. score
average normalized score
fuzzer
aflchurnplusplus 100.00
aflplusplus 93.75
afl 65.62
By avg. rank
average rank
fuzzer
aflchurnplusplus 1.00
aflplusplus 1.07
afl 1.43
  • Critical difference diagram
    The diagram visualizes the average rank of fuzzers (second ranking above) while showing the significance of the differences as well. What is considered a "critical difference" (CD) is based on the Friedman/Nemenyi post-hoc test. See more in the documentation.
    Note: If a fuzzer does not support all benchmarks, its ranking as shown in this diagram can be lower than it should be. So please check the list of supported benchmarks for the fuzzer(s) of your interest. The list could be specified in the fuzzer's README.md like this.
  • Median relative code-coverages on each benchmark

    Note: The relative coverage summary table shows the median relative performance of each fuzzer to the experiment maximum. Thus the highest relative performance may not be 100%.
    trial_relative_coverage = trial_coverage / experiment_max_coverage

      aflchurnplusplus aflplusplus afl
    FuzzerMedian 96.50 96.00 95.00
    FuzzerMean 94.79 93.64 87.08
    arrow_arrow-ipc-stream-fuzz_1a34a0 94.00 96.00 97.00
    aspell_aspell_fuzzer_e8eb74 90.00 89.00 87.00
    assimp_assimp_fuzzer_4d451f 83.00 89.00 50.00
    bloaty_fuzz_target_52948c 96.00 96.00 95.00
    ffmpeg_ffmpeg_demuxer_fuzzer_7adeef 93.00 88.00 66.00
    file_magic_fuzzer_2d5f85 96.00 95.00 nan
    grok_grk_decompress_fuzzer_9cd001 98.00 98.00 96.00
    harfbuzz_hb-shape-fuzzer_17863b 98.00 97.00 99.00
    lcms_cms_transform_all_fuzzer_97d37d 91.00 78.00 63.00
    libaom_av1_dec_fuzzer_6e1848 97.00 97.00 94.00
    libpcap_fuzz_filter_98b0a2 98.00 95.00 93.00
    libxml2_xml_e85b9b 97.00 96.00 98.00
    php_php-fuzz-parser_0dbedb 98.00 99.00 97.00
    systemd_fuzz-network-parser_288baf 98.00 98.00 97.00
    • Fuzzers are sorted by "FuzzerMean" (average median relative coverage), highest on the left.
    • Green background = highest relative median coverage.
    • Blue gradient background = greater than 95% relative median coverage.
  • Median relative bug-coverages on each benchmark

    Note: The relative coverage summary table shows the median relative performance of each fuzzer to the experiment maximum. Thus the highest relative performance may not be 100%.
    trial_relative_coverage = trial_coverage / experiment_max_coverage

      aflchurnplusplus aflplusplus afl
    FuzzerMedian 41.50 41.00 0.00
    FuzzerMean 39.71 38.50 30.54
    arrow_arrow-ipc-stream-fuzz_1a34a0 0.00 0.00 0.00
    aspell_aspell_fuzzer_e8eb74 100.00 100.00 100.00
    assimp_assimp_fuzzer_4d451f 57.00 57.00 14.00
    bloaty_fuzz_target_52948c 100.00 100.00 100.00
    ffmpeg_ffmpeg_demuxer_fuzzer_7adeef 66.00 49.00 0.00
    file_magic_fuzzer_2d5f85 0.00 0.00 nan
    grok_grk_decompress_fuzzer_9cd001 50.00 50.00 50.00
    harfbuzz_hb-shape-fuzzer_17863b 33.00 33.00 33.00
    lcms_cms_transform_all_fuzzer_97d37d 0.00 0.00 0.00
    libaom_av1_dec_fuzzer_6e1848 50.00 50.00 0.00
    libpcap_fuzz_filter_98b0a2 0.00 0.00 0.00
    libxml2_xml_e85b9b 100.00 100.00 100.00
    php_php-fuzz-parser_0dbedb 0.00 0.00 0.00
    systemd_fuzz-network-parser_288baf 0.00 0.00 0.00
    • Fuzzers are sorted by "FuzzerMean" (average median relative coverage), highest on the left.
    • Green background = highest relative median coverage.
    • Blue gradient background = greater than 95% relative median coverage.
  • Total unique bugs found on each benchmark
      Total aflchurnplusplus aflplusplus afl
    FuzzerSum 109 77 77 27
    arrow_arrow-ipc-stream-fuzz_1a34a0 0 0 0 0
    aspell_aspell_fuzzer_e8eb74 2 2 2 2
    assimp_assimp_fuzzer_4d451f 61 38 52 5
    bloaty_fuzz_target_52948c 1 1 1 1
    ffmpeg_ffmpeg_demuxer_fuzzer_7adeef 11 9 6 2
    file_magic_fuzzer_2d5f85 1 1 1 nan
    grok_grk_decompress_fuzzer_9cd001 4 4 2 2
    harfbuzz_hb-shape-fuzzer_17863b 10 4 4 9
    lcms_cms_transform_all_fuzzer_97d37d 2 1 1 0
    libaom_av1_dec_fuzzer_6e1848 9 9 6 4
    libpcap_fuzz_filter_98b0a2 0 0 0 0
    libxml2_xml_e85b9b 2 2 2 2
    php_php-fuzz-parser_0dbedb 1 1 0 0
    systemd_fuzz-network-parser_288baf 5 5 0 0
    • Fuzzers are sorted by "FuzzerSum", highest on the left.
    • Green background = most unique bugs found.
    • *note: This table represents unique bugs found across all trials.

arrow_arrow-ipc-stream-fuzz_1a34a0 summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 14.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    aflchurnplusplus 82800 12.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    aflplusplus 82800 15.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 14.0 2433.428571 37.668012 2330.0 2428.00 2436.0 2450.75 2502.0
    aflplusplus 82800 15.0 2408.933333 49.853309 2293.0 2394.50 2418.0 2441.50 2465.0
    aflchurnplusplus 82800 12.0 2375.083333 63.181497 2302.0 2319.75 2357.5 2432.75 2464.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

aspell_aspell_fuzzer_e8eb74 summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
error
The following fuzzers do not have enough samples: aflchurnplusplus, aflplusplus.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 15.0 1.0 0.0 1.0 1.0 1.0 1.0 1.0
    aflchurnplusplus 82800 10.0 1.0 0.0 1.0 1.0 1.0 1.0 1.0
    aflplusplus 82800 10.0 1.0 0.0 1.0 1.0 1.0 1.0 1.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflchurnplusplus 82800 10.0 3264.200000 135.315927 3084.0 3169.00 3243.0 3311.75 3524.0
    aflplusplus 82800 10.0 3231.400000 122.306718 3155.0 3188.25 3195.5 3208.75 3575.0
    afl 82800 15.0 3132.333333 9.431457 3117.0 3125.00 3136.0 3138.50 3147.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

assimp_assimp_fuzzer_4d451f summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
error
The following fuzzers do not have enough samples: aflplusplus.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflchurnplusplus 82800 13.0 4.692308 1.315587 3.0 4.0 4.0 5.0 7.0
    aflplusplus 82800 11.0 4.090909 1.044466 2.0 4.0 4.0 4.5 6.0
    afl 82800 15.0 1.066667 0.961150 0.0 0.0 1.0 2.0 3.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus 82800 11.0 2695.909091 189.410905 2491.0 2503.0 2705.0 2812.5 3006.0
    aflchurnplusplus 82800 13.0 2588.384615 181.368198 2311.0 2470.0 2505.0 2722.0 2868.0
    afl 82800 15.0 1554.600000 113.364646 1401.0 1463.5 1520.0 1642.5 1740.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

bloaty_fuzz_target_52948c summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 13.0 1.000000 0.000000 1.0 1.00 1.0 1.0 1.0
    aflchurnplusplus 82800 15.0 0.933333 0.258199 0.0 1.00 1.0 1.0 1.0
    aflplusplus 82800 14.0 0.714286 0.468807 0.0 0.25 1.0 1.0 1.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus 82800 14.0 5943.928571 93.901310 5797.0 5870.75 5946.0 6011.5 6132.0
    aflchurnplusplus 82800 15.0 5893.533333 60.092627 5812.0 5854.00 5895.0 5904.5 6010.0
    afl 82800 13.0 5900.615385 123.409574 5652.0 5822.00 5878.0 6007.0 6066.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

ffmpeg_ffmpeg_demuxer_fuzzer_7adeef summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflchurnplusplus 82800 13.0 1.769231 0.725011 1.0 1.0 2.0 2.0 3.0
    aflplusplus 82800 12.0 1.500000 0.522233 1.0 1.0 1.5 2.0 2.0
    afl 82800 12.0 0.500000 0.674200 0.0 0.0 0.0 1.0 2.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflchurnplusplus 82800 13.0 20020.076923 1073.945876 17823.0 19322.00 20062.0 20872.00 21527.0
    aflplusplus 82800 12.0 18878.333333 895.518269 17025.0 18730.75 19033.0 19279.00 20243.0
    afl 82800 12.0 14286.583333 239.239499 13851.0 14131.50 14351.0 14437.75 14670.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

file_magic_fuzzer_2d5f85 summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflchurnplusplus 82800 14.0 0.142857 0.363137 0.0 0.0 0.0 0.0 1.0
    aflplusplus 82800 12.0 0.083333 0.288675 0.0 0.0 0.0 0.0 1.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflchurnplusplus 82800 14.0 2182.214286 85.639835 2041.0 2119.75 2221.0 2230.0 2303.0
    aflplusplus 82800 12.0 2177.750000 69.094303 2045.0 2150.25 2207.5 2227.5 2238.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

grok_grk_decompress_fuzzer_9cd001 summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
error
The following fuzzers do not have enough samples: aflplusplus.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 15.0 1.000000 0.000000 1.0 1.0 1.0 1.0 1.0
    aflchurnplusplus 82800 14.0 1.214286 0.425815 1.0 1.0 1.0 1.0 2.0
    aflplusplus 82800 9.0 1.000000 0.000000 1.0 1.0 1.0 1.0 1.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus 82800 9.0 5882.000000 82.672547 5701.0 5861.00 5874.0 5950.0 5974.0
    aflchurnplusplus 82800 14.0 5818.785714 132.909558 5379.0 5838.25 5858.5 5874.5 5899.0
    afl 82800 15.0 5788.133333 85.320293 5553.0 5768.00 5778.0 5861.5 5889.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

harfbuzz_hb-shape-fuzzer_17863b summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 14.0 1.142857 0.534522 1.0 1.0 1.0 1.0 3.0
    aflchurnplusplus 82800 13.0 1.000000 0.000000 1.0 1.0 1.0 1.0 1.0
    aflplusplus 82800 12.0 1.000000 0.000000 1.0 1.0 1.0 1.0 1.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 14.0 10145.857143 37.292724 10082.0 10121.75 10138.5 10179.5 10201.0
    aflchurnplusplus 82800 13.0 10067.307692 48.866117 9992.0 10024.00 10046.0 10112.0 10132.0
    aflplusplus 82800 12.0 9994.500000 85.287215 9841.0 9946.25 9995.5 10061.5 10115.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

lcms_cms_transform_all_fuzzer_97d37d summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
error
The following fuzzers do not have enough samples: aflchurnplusplus, afl.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 8.0 0.0000 0.000000 0.0 0.0 0.0 0.0 0.0
    aflchurnplusplus 82800 10.0 0.1000 0.316228 0.0 0.0 0.0 0.0 1.0
    aflplusplus 82800 16.0 0.0625 0.250000 0.0 0.0 0.0 0.0 1.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflchurnplusplus 82800 10.0 2149.3000 183.458170 1762.0 2076.00 2183.5 2255.00 2385.0
    aflplusplus 82800 16.0 1835.8125 266.284614 1030.0 1709.25 1862.0 2037.25 2123.0
    afl 82800 8.0 1462.1250 273.624580 1123.0 1215.00 1523.5 1604.50 1915.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

libaom_av1_dec_fuzzer_6e1848 summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflchurnplusplus 82800 12.0 1.000000 0.426401 0.0 1.0 1.0 1.0 2.0
    aflplusplus 82800 11.0 0.636364 0.674200 0.0 0.0 1.0 1.0 2.0
    afl 82800 11.0 0.454545 0.522233 0.0 0.0 0.0 1.0 1.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus 82800 11.0 10517.181818 99.277206 10329.0 10463.50 10511.0 10587.00 10673.0
    aflchurnplusplus 82800 12.0 10483.083333 122.795809 10314.0 10430.75 10469.0 10520.25 10767.0
    afl 82800 11.0 10222.636364 115.091505 10091.0 10135.00 10186.0 10297.50 10420.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

libpcap_fuzz_filter_98b0a2 summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
error
The following fuzzers do not have enough samples: aflchurnplusplus.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 13.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    aflchurnplusplus 82800 10.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    aflplusplus 82800 11.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflchurnplusplus 82800 10.0 3452.200000 72.298455 3328.0 3402.25 3482.5 3513.25 3524.0
    aflplusplus 82800 11.0 3362.545455 52.813566 3291.0 3329.00 3365.0 3375.50 3480.0
    afl 82800 13.0 3324.000000 32.298091 3284.0 3299.00 3310.0 3355.00 3373.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

libxml2_xml_e85b9b summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 12.0 1.000000 0.000000 1.0 1.0 1.0 1.0 1.0
    aflchurnplusplus 82800 13.0 0.769231 0.438529 0.0 1.0 1.0 1.0 1.0
    aflplusplus 82800 14.0 0.642857 0.497245 0.0 0.0 1.0 1.0 1.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 12.0 19683.583333 171.210906 19484.0 19559.25 19657.5 19752.00 20027.0
    aflchurnplusplus 82800 13.0 19450.692308 138.197555 19173.0 19361.00 19455.0 19543.00 19676.0
    aflplusplus 82800 14.0 19325.500000 257.931043 18854.0 19222.25 19323.0 19442.75 19884.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

php_php-fuzz-parser_0dbedb summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
error
The following fuzzers do not have enough samples: aflchurnplusplus, afl.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 10.0 0.000000 0.00000 0.0 0.0 0.0 0.0 0.0
    aflchurnplusplus 82800 11.0 0.181818 0.40452 0.0 0.0 0.0 0.0 1.0
    aflplusplus 82800 15.0 0.000000 0.00000 0.0 0.0 0.0 0.0 0.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus 82800 15.0 16896.266667 92.789675 16740.0 16843.50 16906.0 16940.5 17050.0
    aflchurnplusplus 82800 11.0 16872.454545 110.395076 16757.0 16774.00 16804.0 16950.5 17039.0
    afl 82800 10.0 16630.600000 47.905462 16587.0 16597.75 16614.0 16649.5 16746.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

systemd_fuzz-network-parser_288baf summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 12.0 0.00 0.000000 0.0 0.0 0.0 0.0 0.0
    aflchurnplusplus 82800 12.0 0.25 0.866025 0.0 0.0 0.0 0.0 3.0
    aflplusplus 82800 14.0 0.00 0.000000 0.0 0.0 0.0 0.0 0.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflchurnplusplus 82800 12.0 3405.250000 29.086470 3364.0 3388.75 3398.5 3428.50 3456.0
    aflplusplus 82800 14.0 3396.357143 28.272322 3350.0 3381.25 3394.0 3415.25 3452.0
    afl 82800 12.0 3385.000000 27.352912 3339.0 3366.75 3384.0 3406.75 3423.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

experiment data

You can download the raw data for this report here.

Check out the documentation on how to create customized reports using this data. Also see some example Colab notebooks for doing custom analysis on the data here.

Experiment Description:

(None,)