FuzzBench: 2023-05-09-new-aflpp-bug report

experiment summary

We show two different aggregate (cross-benchmark) rankings of fuzzers. The first is based on the average of per-benchmarks scores, where the score represents the percentage of the highest reached median bug-coverage on a given benchmark (higher value is better). The second ranking shows the average rank of fuzzers, after we rank them on each benchmark according to their median reached bug-covereges (lower value is better).
By avg. score
average normalized score
fuzzer
aflpp_d82218 75.93
aflppp_d82218 75.93
pastis 70.37
aflrustrust 70.37
aflsmart_plusplus 68.52
afl 62.96
honggfuzz 61.11
aflplusplusplus 51.85
hastefuzz 51.85
libfuzzer 51.85
aflplusplus 48.15
By avg. rank
average rank
fuzzer
aflpp_d82218 1.71
aflppp_d82218 1.71
aflrustrust 2.14
honggfuzz 2.21
aflsmart_plusplus 2.29
pastis 2.43
afl 2.71
hastefuzz 2.86
aflplusplusplus 2.93
libfuzzer 2.93
aflplusplus 3.14
  • Critical difference diagram
    The diagram visualizes the average rank of fuzzers (second ranking above) while showing the significance of the differences as well. What is considered a "critical difference" (CD) is based on the Friedman/Nemenyi post-hoc test. See more in the documentation.
    Note: If a fuzzer does not support all benchmarks, its ranking as shown in this diagram can be lower than it should be. So please check the list of supported benchmarks for the fuzzer(s) of your interest. The list could be specified in the fuzzer's README.md like this.
  • Median relative code-coverages on each benchmark

    Note: The relative coverage summary table shows the median relative performance of each fuzzer to the experiment maximum. Thus the highest relative performance may not be 100%.
    trial_relative_coverage = trial_coverage / experiment_max_coverage

      aflppp_d82218 aflpp_d82218 honggfuzz aflplusplusplus hastefuzz aflrustrust aflplusplus pastis afl aflsmart_plusplus libfuzzer
    FuzzerMedian 95.00 94.00 87.00 89.50 89.50 92.00 89.00 84.50 93.00 92.50 86.00
    FuzzerMean 90.92 90.15 86.00 85.71 85.43 85.21 83.50 83.25 83.00 82.64 82.00
    aspell_aspell_fuzzer_e8eb74 83.00 78.00 81.00 77.00 79.00 79.00 77.00 76.00 77.00 77.00 75.00
    assimp_assimp_fuzzer_4d451f 76.00 77.00 87.00 68.00 64.00 55.00 55.00 77.00 38.00 40.00 76.00
    bloaty_fuzz_target_52948c 90.00 90.00 91.00 75.00 79.00 94.00 89.00 85.00 96.00 95.00 nan
    ffmpeg_ffmpeg_demuxer_fuzzer_7adeef 89.00 88.00 83.00 84.00 62.00 57.00 74.00 84.00 60.00 60.00 66.00
    file_magic_fuzzer_2d5f85 89.00 89.00 nan 89.00 98.00 99.00 89.00 73.00 93.00 92.00 92.00
    grok_grk_decompress_fuzzer_9cd001 97.00 97.00 94.00 96.00 96.00 93.00 95.00 96.00 95.00 94.00 92.00
    harfbuzz_hb-shape-fuzzer_17863b 97.00 97.00 98.00 97.00 98.00 91.00 97.00 nan 98.00 98.00 86.00
    lcms_cms_transform_all_fuzzer_97d37d 78.00 78.00 72.00 78.00 88.00 83.00 60.00 77.00 64.00 61.00 74.00
    libaom_av1_dec_fuzzer_6e1848 98.00 98.00 97.00 93.00 97.00 96.00 94.00 97.00 94.00 96.00 90.00
    libpcap_fuzz_filter_98b0a2 97.00 94.00 90.00 96.00 94.00 90.00 94.00 94.00 92.00 88.00 86.00
    libxml2_xml_e85b9b 95.00 96.00 85.00 93.00 91.00 95.00 95.00 98.00 98.00 98.00 76.00
    mbedtls_fuzz_dtlsclient_7c6b0e nan nan 68.00 68.00 69.00 68.00 68.00 nan 68.00 69.00 68.00
    php_php-fuzz-parser_0dbedb 96.00 96.00 98.00 96.00 95.00 95.00 96.00 98.00 96.00 96.00 95.00
    systemd_fuzz-network-parser_288baf 97.00 94.00 74.00 90.00 86.00 98.00 86.00 44.00 93.00 93.00 90.00
    • Fuzzers are sorted by "FuzzerMean" (average median relative coverage), highest on the left.
    • Green background = highest relative median coverage.
    • Blue gradient background = greater than 95% relative median coverage.
  • Median relative bug-coverages on each benchmark

    Note: The relative coverage summary table shows the median relative performance of each fuzzer to the experiment maximum. Thus the highest relative performance may not be 100%.
    trial_relative_coverage = trial_coverage / experiment_max_coverage

      pastis aflpp_d82218 aflppp_d82218 aflrustrust aflsmart_plusplus honggfuzz afl libfuzzer hastefuzz aflplusplusplus aflplusplus
    FuzzerMedian 41.50 50.00 50.00 27.50 22.00 33.00 5.50 0.00 0.00 0.00 0.00
    FuzzerMean 37.83 36.85 34.92 28.93 28.14 25.54 24.57 20.00 19.79 18.57 17.00
    aspell_aspell_fuzzer_e8eb74 66.00 33.00 33.00 33.00 33.00 33.00 33.00 33.00 33.00 33.00 33.00
    assimp_assimp_fuzzer_4d451f 55.00 55.00 55.00 22.00 11.00 66.00 11.00 44.00 44.00 44.00 22.00
    bloaty_fuzz_target_52948c 100.00 100.00 100.00 100.00 100.00 0.00 100.00 nan 0.00 0.00 0.00
    ffmpeg_ffmpeg_demuxer_fuzzer_7adeef 33.00 66.00 66.00 0.00 0.00 33.00 0.00 33.00 0.00 33.00 33.00
    file_magic_fuzzer_2d5f85 0.00 25.00 0.00 50.00 50.00 nan 50.00 50.00 50.00 50.00 50.00
    grok_grk_decompress_fuzzer_9cd001 50.00 50.00 50.00 50.00 50.00 50.00 50.00 50.00 50.00 50.00 50.00
    harfbuzz_hb-shape-fuzzer_17863b nan 50.00 50.00 50.00 50.00 50.00 50.00 50.00 50.00 50.00 50.00
    lcms_cms_transform_all_fuzzer_97d37d 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
    libaom_av1_dec_fuzzer_6e1848 100.00 50.00 50.00 50.00 50.00 50.00 0.00 0.00 50.00 0.00 0.00
    libpcap_fuzz_filter_98b0a2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
    libxml2_xml_e85b9b 50.00 50.00 50.00 50.00 50.00 50.00 50.00 0.00 0.00 0.00 0.00
    mbedtls_fuzz_dtlsclient_7c6b0e nan nan nan 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
    php_php-fuzz-parser_0dbedb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
    systemd_fuzz-network-parser_288baf 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
    • Fuzzers are sorted by "FuzzerMean" (average median relative coverage), highest on the left.
    • Green background = highest relative median coverage.
    • Blue gradient background = greater than 95% relative median coverage.
  • Total unique bugs found on each benchmark
      Total honggfuzz pastis aflppp_d82218 aflpp_d82218 aflplusplusplus hastefuzz aflrustrust aflplusplus afl libfuzzer aflsmart_plusplus
    FuzzerSum 226 133 101 100 94 84 75 70 45 39 39 37
    aspell_aspell_fuzzer_e8eb74 5 2 5 2 2 2 2 2 2 2 2 2
    assimp_assimp_fuzzer_4d451f 144 83 61 56 52 55 44 30 20 3 24 2
    bloaty_fuzz_target_52948c 1 1 1 1 1 0 1 1 1 1 0 1
    ffmpeg_ffmpeg_demuxer_fuzzer_7adeef 19 7 7 13 13 9 4 3 7 5 2 2
    file_magic_fuzzer_2d5f85 3 nan 0 1 1 1 2 1 1 3 1 2
    grok_grk_decompress_fuzzer_9cd001 6 2 2 4 3 3 2 2 2 2 2 3
    harfbuzz_hb-shape-fuzzer_17863b 8 5 4 6 3 7 5 5 4 5 3 6
    lcms_cms_transform_all_fuzzer_97d37d 8 2 2 0 2 2 2 1 0 1 3 0
    libaom_av1_dec_fuzzer_6e1848 16 16 16 14 15 3 13 16 6 12 0 15
    libpcap_fuzz_filter_98b0a2 0 0 0 0 0 0 0 0 0 0 0 0
    libxml2_xml_e85b9b 3 2 2 2 2 1 0 2 1 3 2 3
    mbedtls_fuzz_dtlsclient_7c6b0e 0 0 0 nan nan 0 0 0 0 0 0 0
    php_php-fuzz-parser_0dbedb 3 3 1 1 0 1 0 0 1 2 0 1
    systemd_fuzz-network-parser_288baf 10 10 0 0 0 0 0 7 0 0 0 0
    • Fuzzers are sorted by "FuzzerSum", highest on the left.
    • Green background = most unique bugs found.
    • *note: This table represents unique bugs found across all trials.

aspell_aspell_fuzzer_e8eb74 summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    pastis 82800 20.0 2.10 0.307794 2.0 2.0 2.0 2.0 3.0
    afl 82800 20.0 1.00 0.000000 1.0 1.0 1.0 1.0 1.0
    aflplusplus 82800 20.0 1.00 0.000000 1.0 1.0 1.0 1.0 1.0
    aflplusplusplus 82800 20.0 0.90 0.307794 0.0 1.0 1.0 1.0 1.0
    aflpp_d82218 82800 20.0 1.00 0.000000 1.0 1.0 1.0 1.0 1.0
    aflppp_d82218 82800 20.0 1.00 0.000000 1.0 1.0 1.0 1.0 1.0
    aflrustrust 82800 20.0 0.85 0.366348 0.0 1.0 1.0 1.0 1.0
    aflsmart_plusplus 82800 20.0 1.00 0.000000 1.0 1.0 1.0 1.0 1.0
    hastefuzz 82800 20.0 0.70 0.470162 0.0 0.0 1.0 1.0 1.0
    honggfuzz 82800 20.0 0.75 0.550120 0.0 0.0 1.0 1.0 2.0
    libfuzzer 82800 20.0 0.80 0.410391 0.0 1.0 1.0 1.0 1.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflppp_d82218 82800 20.0 3459.00 277.468822 3145.0 3205.50 3418.5 3734.25 3908.0
    honggfuzz 82800 20.0 3322.95 69.073169 3246.0 3275.00 3304.5 3333.25 3533.0
    hastefuzz 82800 20.0 3309.65 270.971017 2986.0 3140.50 3227.5 3448.50 3957.0
    aflrustrust 82800 20.0 3216.85 92.621059 3029.0 3152.50 3227.0 3276.50 3356.0
    aflpp_d82218 82800 20.0 3341.80 202.578796 3153.0 3183.75 3213.5 3523.50 3867.0
    aflplusplusplus 82800 20.0 3290.45 279.587944 2961.0 3132.50 3171.0 3427.75 4072.0
    aflplusplus 82800 20.0 3248.30 173.044290 3135.0 3156.00 3169.5 3252.75 3812.0
    afl 82800 20.0 3139.55 7.104298 3116.0 3137.00 3140.5 3142.25 3156.0
    aflsmart_plusplus 82800 20.0 3139.75 6.835472 3122.0 3137.25 3139.5 3142.50 3154.0
    pastis 82800 20.0 3151.00 62.793731 3076.0 3112.00 3125.0 3196.00 3295.0
    libfuzzer 82800 20.0 3068.00 80.004605 2930.0 3005.75 3055.0 3124.00 3222.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

assimp_assimp_fuzzer_4d451f summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    honggfuzz 82800 20.0 6.300000 1.490320 4.0 5.00 6.0 7.25 9.0
    aflpp_d82218 82800 18.0 5.666667 1.371989 4.0 5.00 5.0 7.00 8.0
    aflppp_d82218 82800 20.0 5.200000 1.239694 3.0 4.00 5.0 6.00 8.0
    pastis 82800 20.0 5.600000 1.095445 4.0 5.00 5.0 6.00 8.0
    aflplusplusplus 82800 17.0 4.176471 1.286239 2.0 4.00 4.0 5.00 7.0
    hastefuzz 82800 19.0 4.210526 1.134262 2.0 3.50 4.0 5.00 7.0
    libfuzzer 82800 20.0 4.450000 1.145931 3.0 3.75 4.0 5.25 6.0
    aflplusplus 82800 18.0 2.111111 0.900254 1.0 2.00 2.0 2.00 4.0
    aflrustrust 82800 20.0 1.950000 1.276302 0.0 1.00 2.0 3.00 4.0
    afl 82800 20.0 1.050000 0.825578 0.0 0.75 1.0 1.25 3.0
    aflsmart_plusplus 82800 20.0 0.700000 0.656947 0.0 0.00 1.0 1.00 2.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    honggfuzz 82800 20.0 3396.350000 196.457113 3143.0 3245.25 3396.0 3501.00 3885.0
    pastis 82800 20.0 3020.600000 191.356487 2691.0 2929.00 3027.5 3151.00 3505.0
    aflpp_d82218 82800 18.0 3026.388889 253.644758 2593.0 2894.00 3007.5 3178.25 3491.0
    libfuzzer 82800 20.0 2979.950000 119.598220 2712.0 2932.75 2981.0 3060.50 3178.0
    aflppp_d82218 82800 20.0 2995.750000 217.054656 2710.0 2812.25 2973.0 3160.25 3461.0
    aflplusplusplus 82800 17.0 2664.882353 230.741978 2287.0 2495.00 2659.0 2774.00 3073.0
    hastefuzz 82800 19.0 2540.263158 180.687835 2273.0 2428.00 2516.0 2606.00 2949.0
    aflrustrust 82800 20.0 2115.500000 273.123165 1689.0 1854.25 2151.0 2375.00 2485.0
    aflplusplus 82800 18.0 2158.166667 117.732623 1941.0 2088.00 2148.5 2247.00 2376.0
    aflsmart_plusplus 82800 20.0 1525.050000 92.491522 1315.0 1491.50 1554.0 1577.50 1670.0
    afl 82800 20.0 1538.250000 108.713953 1369.0 1463.00 1514.5 1611.00 1749.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

bloaty_fuzz_target_52948c summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 0.95 0.223607 0.0 1.0 1.0 1.0 1.0
    aflpp_d82218 82800 20.0 0.90 0.307794 0.0 1.0 1.0 1.0 1.0
    aflppp_d82218 82800 20.0 0.90 0.307794 0.0 1.0 1.0 1.0 1.0
    aflrustrust 82800 20.0 1.00 0.000000 1.0 1.0 1.0 1.0 1.0
    aflsmart_plusplus 82800 16.0 1.00 0.000000 1.0 1.0 1.0 1.0 1.0
    pastis 82800 20.0 0.80 0.410391 0.0 1.0 1.0 1.0 1.0
    aflplusplus 82800 20.0 0.40 0.502625 0.0 0.0 0.0 1.0 1.0
    aflplusplusplus 82800 20.0 0.00 0.000000 0.0 0.0 0.0 0.0 0.0
    hastefuzz 82800 20.0 0.15 0.366348 0.0 0.0 0.0 0.0 1.0
    honggfuzz 82800 20.0 0.40 0.502625 0.0 0.0 0.0 1.0 1.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 5947.8500 134.426765 5636.0 5913.50 5995.0 6033.50 6085.0
    aflsmart_plusplus 82800 16.0 5929.8125 143.821287 5676.0 5884.25 5938.5 5993.00 6220.0
    aflrustrust 82800 20.0 5864.4000 89.901349 5667.0 5818.75 5863.5 5919.50 6030.0
    honggfuzz 82800 20.0 5765.4500 141.435674 5598.0 5663.00 5712.5 5833.75 6077.0
    aflpp_d82218 82800 20.0 5715.9500 183.197183 5442.0 5586.50 5659.0 5892.00 6051.0
    aflppp_d82218 82800 20.0 5653.1000 151.682459 5346.0 5545.75 5625.0 5776.75 5898.0
    aflplusplus 82800 20.0 5544.0500 84.289898 5386.0 5480.25 5542.5 5609.00 5699.0
    pastis 82800 20.0 5283.2500 113.848410 5042.0 5190.00 5304.0 5359.25 5495.0
    hastefuzz 82800 20.0 4955.3500 158.122629 4689.0 4826.50 4925.5 5108.25 5203.0
    aflplusplusplus 82800 20.0 4688.3000 89.453958 4538.0 4616.00 4681.0 4746.50 4867.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

ffmpeg_ffmpeg_demuxer_fuzzer_7adeef summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflpp_d82218 82800 20.0 1.850000 0.670820 1.0 1.0 2.0 2.00 3.0
    aflppp_d82218 82800 20.0 1.850000 0.670820 1.0 1.0 2.0 2.00 3.0
    aflplusplus 82800 20.0 0.950000 0.223607 0.0 1.0 1.0 1.00 1.0
    aflplusplusplus 82800 20.0 1.400000 0.753937 0.0 1.0 1.0 2.00 3.0
    honggfuzz 82800 20.0 1.250000 0.444262 1.0 1.0 1.0 1.25 2.0
    libfuzzer 82800 20.0 0.900000 0.307794 0.0 1.0 1.0 1.00 1.0
    pastis 82800 19.0 1.157895 0.374634 1.0 1.0 1.0 1.00 2.0
    afl 82800 20.0 0.350000 0.489360 0.0 0.0 0.0 1.00 1.0
    aflrustrust 82800 20.0 0.250000 0.716350 0.0 0.0 0.0 0.00 3.0
    aflsmart_plusplus 82800 19.0 0.263158 0.452414 0.0 0.0 0.0 0.50 1.0
    hastefuzz 82800 20.0 0.250000 0.444262 0.0 0.0 0.0 0.25 1.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflppp_d82218 82800 20.0 21164.650000 835.968853 19108.0 20705.75 21187.0 21572.75 22766.0
    aflpp_d82218 82800 20.0 21139.450000 1211.346118 18483.0 20669.25 20991.5 22033.00 23663.0
    aflplusplusplus 82800 20.0 19953.250000 1283.997126 16290.0 19551.75 20065.5 20596.50 21926.0
    pastis 82800 19.0 19999.789474 429.057828 19112.0 19718.50 20024.0 20292.00 20783.0
    honggfuzz 82800 20.0 19726.800000 612.479998 18154.0 19625.25 19867.0 20104.75 20535.0
    aflplusplus 82800 20.0 17609.900000 557.859056 16648.0 17274.00 17618.5 17843.25 18660.0
    libfuzzer 82800 20.0 15740.050000 370.563792 14835.0 15579.25 15769.5 16000.75 16270.0
    hastefuzz 82800 20.0 15109.950000 1025.964372 13026.0 14515.75 14860.0 15715.25 17269.0
    aflsmart_plusplus 82800 19.0 14278.894737 369.709780 13559.0 14106.00 14363.0 14482.50 14958.0
    afl 82800 20.0 14210.000000 486.142712 13046.0 13831.25 14307.0 14565.50 14909.0
    aflrustrust 82800 20.0 13352.550000 3849.710987 4372.0 11431.50 13586.0 16452.00 18781.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

file_magic_fuzzer_2d5f85 summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 1.050000 0.223607 1.0 1.00 1.0 1.0 2.0
    aflplusplus 82800 20.0 0.750000 0.444262 0.0 0.75 1.0 1.0 1.0
    aflplusplusplus 82800 20.0 0.600000 0.502625 0.0 0.00 1.0 1.0 1.0
    aflrustrust 82800 20.0 1.000000 0.000000 1.0 1.00 1.0 1.0 1.0
    aflsmart_plusplus 82800 18.0 1.111111 0.323381 1.0 1.00 1.0 1.0 2.0
    hastefuzz 82800 17.0 1.000000 0.000000 1.0 1.00 1.0 1.0 1.0
    libfuzzer 82800 20.0 1.000000 0.000000 1.0 1.00 1.0 1.0 1.0
    aflpp_d82218 82800 20.0 0.500000 0.512989 0.0 0.00 0.5 1.0 1.0
    aflppp_d82218 82800 20.0 0.450000 0.510418 0.0 0.00 0.0 1.0 1.0
    pastis 82800 20.0 0.000000 0.000000 0.0 0.00 0.0 0.0 0.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflrustrust 82800 20.0 2496.400000 12.542139 2472.0 2485.50 2498.0 2504.25 2521.0
    hastefuzz 82800 17.0 2480.411765 22.383194 2437.0 2465.00 2485.0 2499.00 2511.0
    afl 82800 20.0 2360.450000 11.997697 2337.0 2351.75 2358.0 2371.25 2379.0
    aflsmart_plusplus 82800 18.0 2341.166667 14.857163 2321.0 2332.00 2334.5 2348.50 2375.0
    libfuzzer 82800 20.0 2331.700000 8.163462 2319.0 2325.75 2329.5 2338.75 2344.0
    aflppp_d82218 82800 20.0 2233.850000 71.373204 2020.0 2214.50 2251.5 2257.25 2403.0
    aflpp_d82218 82800 20.0 2238.650000 59.043494 2071.0 2224.75 2249.0 2260.25 2355.0
    aflplusplusplus 82800 20.0 2256.400000 71.450682 2083.0 2234.00 2246.5 2273.00 2391.0
    aflplusplus 82800 20.0 2252.400000 63.242557 2159.0 2219.50 2246.0 2269.50 2393.0
    pastis 82800 20.0 1851.450000 4.248529 1844.0 1848.75 1852.0 1854.25 1858.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

grok_grk_decompress_fuzzer_9cd001 summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
error
The following fuzzers do not have enough samples: aflsmart_plusplus, pastis.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 1.00 0.000000 1.0 1.0 1.0 1.0 1.0
    aflplusplus 82800 20.0 1.00 0.000000 1.0 1.0 1.0 1.0 1.0
    aflplusplusplus 82800 20.0 1.05 0.223607 1.0 1.0 1.0 1.0 2.0
    aflpp_d82218 82800 20.0 1.05 0.223607 1.0 1.0 1.0 1.0 2.0
    aflppp_d82218 82800 20.0 1.15 0.366348 1.0 1.0 1.0 1.0 2.0
    aflrustrust 82800 20.0 1.00 0.000000 1.0 1.0 1.0 1.0 1.0
    aflsmart_plusplus 82800 9.0 1.00 0.000000 1.0 1.0 1.0 1.0 1.0
    hastefuzz 82800 20.0 1.00 0.000000 1.0 1.0 1.0 1.0 1.0
    honggfuzz 82800 20.0 1.00 0.000000 1.0 1.0 1.0 1.0 1.0
    libfuzzer 82800 20.0 1.00 0.000000 1.0 1.0 1.0 1.0 1.0
    pastis 82800 2.0 1.00 0.000000 1.0 1.0 1.0 1.0 1.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflppp_d82218 82800 20.0 6230.600000 62.925520 6103.0 6180.25 6223.5 6278.50 6336.0
    aflpp_d82218 82800 20.0 6216.800000 66.920691 6054.0 6190.00 6216.0 6243.75 6407.0
    pastis 82800 2.0 6203.500000 33.234019 6180.0 6191.75 6203.5 6215.25 6227.0
    aflplusplusplus 82800 20.0 6195.300000 53.381546 6110.0 6151.75 6178.5 6239.25 6282.0
    hastefuzz 82800 20.0 6142.300000 77.384957 5961.0 6094.25 6153.5 6191.25 6253.0
    aflplusplus 82800 20.0 6145.800000 74.560184 5994.0 6119.25 6133.5 6172.00 6391.0
    afl 82800 20.0 6089.000000 55.986841 6007.0 6047.50 6088.5 6128.25 6189.0
    honggfuzz 82800 20.0 6089.350000 24.602685 6040.0 6073.00 6084.0 6108.25 6128.0
    aflsmart_plusplus 82800 9.0 6070.333333 63.255435 5998.0 6025.00 6060.0 6092.00 6166.0
    aflrustrust 82800 20.0 5978.650000 154.930978 5518.0 5927.25 5997.5 6086.50 6204.0
    libfuzzer 82800 20.0 5943.100000 40.895116 5887.0 5916.50 5943.5 5966.75 6037.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

harfbuzz_hb-shape-fuzzer_17863b summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 1.25 0.444262 1.0 1.0 1.0 1.25 2.0
    aflplusplus 82800 20.0 1.00 0.000000 1.0 1.0 1.0 1.00 1.0
    aflplusplusplus 82800 20.0 1.05 0.394034 0.0 1.0 1.0 1.00 2.0
    aflpp_d82218 82800 20.0 1.00 0.000000 1.0 1.0 1.0 1.00 1.0
    aflppp_d82218 82800 20.0 1.05 0.223607 1.0 1.0 1.0 1.00 2.0
    aflrustrust 82800 20.0 0.85 0.670820 0.0 0.0 1.0 1.00 2.0
    aflsmart_plusplus 82800 20.0 1.10 0.307794 1.0 1.0 1.0 1.00 2.0
    hastefuzz 82800 20.0 1.00 0.000000 1.0 1.0 1.0 1.00 1.0
    honggfuzz 82800 20.0 1.00 0.000000 1.0 1.0 1.0 1.00 1.0
    libfuzzer 82800 20.0 1.00 0.000000 1.0 1.0 1.0 1.00 1.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflsmart_plusplus 82800 20.0 10178.85 45.617719 10096.0 10148.50 10191.0 10199.50 10284.0
    afl 82800 20.0 10186.85 38.955136 10099.0 10158.50 10180.0 10215.00 10262.0
    hastefuzz 82800 20.0 10159.65 81.822192 9872.0 10136.75 10174.0 10200.25 10261.0
    honggfuzz 82800 20.0 10150.50 50.514719 10077.0 10099.50 10160.0 10191.25 10227.0
    aflplusplus 82800 20.0 10048.45 125.212020 9658.0 10032.75 10084.0 10120.00 10159.0
    aflplusplusplus 82800 20.0 9961.35 440.676069 8138.0 10040.00 10075.5 10107.50 10157.0
    aflppp_d82218 82800 20.0 10053.05 59.865177 9926.0 10013.50 10064.0 10085.75 10179.0
    aflpp_d82218 82800 20.0 10040.55 49.898291 9945.0 10001.75 10041.5 10082.75 10111.0
    aflrustrust 82800 20.0 9107.00 754.933946 7910.0 8341.00 9427.0 9618.75 10302.0
    libfuzzer 82800 20.0 8950.80 134.508697 8672.0 8875.00 8947.5 9031.50 9240.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

lcms_cms_transform_all_fuzzer_97d37d summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 0.050000 0.223607 0.0 0.0 0.0 0.0 1.0
    aflplusplus 82800 20.0 0.000000 0.000000 0.0 0.0 0.0 0.0 0.0
    aflplusplusplus 82800 20.0 0.100000 0.307794 0.0 0.0 0.0 0.0 1.0
    aflpp_d82218 82800 20.0 0.150000 0.366348 0.0 0.0 0.0 0.0 1.0
    aflppp_d82218 82800 20.0 0.000000 0.000000 0.0 0.0 0.0 0.0 0.0
    aflrustrust 82800 20.0 0.400000 0.502625 0.0 0.0 0.0 1.0 1.0
    aflsmart_plusplus 82800 20.0 0.000000 0.000000 0.0 0.0 0.0 0.0 0.0
    hastefuzz 82800 20.0 0.100000 0.447214 0.0 0.0 0.0 0.0 2.0
    honggfuzz 82800 20.0 0.100000 0.307794 0.0 0.0 0.0 0.0 1.0
    libfuzzer 82800 20.0 0.150000 0.366348 0.0 0.0 0.0 0.0 1.0
    pastis 82800 17.0 0.117647 0.332106 0.0 0.0 0.0 0.0 1.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    hastefuzz 82800 20.0 2193.350000 182.744652 1672.0 2108.25 2248.0 2325.75 2417.0
    aflrustrust 82800 20.0 2100.600000 239.607442 1679.0 1896.00 2135.5 2238.25 2531.0
    aflpp_d82218 82800 20.0 2069.650000 197.449334 1836.0 1927.00 2009.0 2164.25 2545.0
    aflppp_d82218 82800 20.0 1915.200000 357.844806 1102.0 1766.50 1996.0 2141.75 2428.0
    aflplusplusplus 82800 20.0 1955.950000 332.059994 1125.0 1879.00 1985.5 2162.00 2437.0
    pastis 82800 17.0 1940.176471 134.148814 1623.0 1906.00 1977.0 2043.00 2105.0
    libfuzzer 82800 20.0 1865.950000 182.349137 1529.0 1731.25 1884.5 1975.75 2162.0
    honggfuzz 82800 20.0 1761.550000 264.876828 1180.0 1569.50 1838.5 1943.25 2181.0
    afl 82800 20.0 1601.450000 278.045524 1017.0 1482.00 1633.5 1764.50 2103.0
    aflsmart_plusplus 82800 20.0 1518.600000 244.655160 991.0 1466.50 1564.0 1687.50 1879.0
    aflplusplus 82800 20.0 1562.650000 189.490251 1282.0 1415.25 1529.0 1671.50 2021.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

libaom_av1_dec_fuzzer_6e1848 summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
error
The following fuzzers do not have enough samples: pastis.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    pastis 82800 5.0 1.600000 0.547723 1.0 1.00 2.0 2.00 2.0
    aflpp_d82218 82800 20.0 1.400000 0.502625 1.0 1.00 1.0 2.00 2.0
    aflppp_d82218 82800 20.0 1.100000 0.640723 0.0 1.00 1.0 1.25 2.0
    aflrustrust 82800 20.0 1.000000 0.725476 0.0 0.75 1.0 1.25 2.0
    aflsmart_plusplus 82800 18.0 1.055556 0.639137 0.0 1.00 1.0 1.00 2.0
    hastefuzz 82800 20.0 1.150000 0.670820 0.0 1.00 1.0 2.00 2.0
    honggfuzz 82800 20.0 1.200000 0.615587 0.0 1.00 1.0 2.00 2.0
    afl 82800 20.0 0.500000 0.606977 0.0 0.00 0.0 1.00 2.0
    aflplusplus 82800 20.0 0.450000 0.510418 0.0 0.00 0.0 1.00 1.0
    aflplusplusplus 82800 20.0 0.250000 0.444262 0.0 0.00 0.0 0.25 1.0
    libfuzzer 82800 20.0 0.000000 0.000000 0.0 0.00 0.0 0.00 0.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflpp_d82218 82800 20.0 10716.300000 46.202187 10619.0 10693.50 10714.0 10725.25 10849.0
    aflppp_d82218 82800 20.0 10618.400000 140.801989 10237.0 10580.50 10667.0 10716.00 10761.0
    hastefuzz 82800 20.0 10559.900000 160.970380 10074.0 10524.50 10613.5 10655.00 10746.0
    honggfuzz 82800 20.0 10557.400000 101.410785 10260.0 10500.25 10578.5 10634.25 10697.0
    pastis 82800 5.0 10531.000000 140.838560 10301.0 10500.00 10575.0 10639.00 10640.0
    aflsmart_plusplus 82800 18.0 10478.111111 102.549781 10281.0 10384.50 10507.5 10568.50 10590.0
    aflrustrust 82800 20.0 10439.150000 162.794438 10138.0 10315.25 10478.0 10526.25 10722.0
    aflplusplus 82800 20.0 10227.200000 186.408606 9860.0 10102.00 10239.0 10341.50 10540.0
    afl 82800 20.0 10219.600000 134.824018 9998.0 10128.50 10198.5 10328.00 10465.0
    aflplusplusplus 82800 20.0 10195.950000 100.478055 10068.0 10108.75 10171.0 10268.25 10358.0
    libfuzzer 82800 20.0 9810.550000 58.383915 9733.0 9764.00 9800.0 9852.50 9928.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

libpcap_fuzz_filter_98b0a2 summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    aflplusplus 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    aflplusplusplus 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    aflpp_d82218 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    aflppp_d82218 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    aflrustrust 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    aflsmart_plusplus 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    hastefuzz 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    honggfuzz 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    libfuzzer 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    pastis 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflppp_d82218 82800 20.0 3474.10 74.212994 3363.0 3413.75 3489.0 3530.75 3587.0
    aflplusplusplus 82800 20.0 3461.65 52.324167 3369.0 3422.75 3462.5 3502.75 3545.0
    aflplusplus 82800 20.0 3405.90 44.441446 3333.0 3366.00 3404.5 3430.75 3497.0
    aflpp_d82218 82800 20.0 3425.05 73.510096 3339.0 3372.75 3397.5 3482.00 3557.0
    hastefuzz 82800 20.0 3397.45 81.283438 3281.0 3327.50 3392.0 3486.50 3506.0
    pastis 82800 20.0 3376.25 45.354597 3290.0 3340.00 3379.0 3400.50 3469.0
    afl 82800 20.0 3325.50 28.426267 3265.0 3307.75 3324.0 3347.25 3376.0
    honggfuzz 82800 20.0 3266.45 27.755464 3223.0 3248.75 3264.0 3282.50 3340.0
    aflrustrust 82800 20.0 3249.90 35.002105 3166.0 3232.25 3258.0 3277.00 3291.0
    aflsmart_plusplus 82800 20.0 3203.90 108.042341 2977.0 3152.50 3188.0 3287.75 3391.0
    libfuzzer 82800 20.0 3093.45 100.218958 2932.0 3002.50 3107.0 3174.00 3260.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

libxml2_xml_e85b9b summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 1.10 0.307794 1.0 1.00 1.0 1.00 2.0
    aflpp_d82218 82800 20.0 0.90 0.307794 0.0 1.00 1.0 1.00 1.0
    aflppp_d82218 82800 20.0 0.70 0.470162 0.0 0.00 1.0 1.00 1.0
    aflrustrust 82800 20.0 0.75 0.444262 0.0 0.75 1.0 1.00 1.0
    aflsmart_plusplus 82800 20.0 1.05 0.394034 0.0 1.00 1.0 1.00 2.0
    honggfuzz 82800 20.0 0.75 0.444262 0.0 0.75 1.0 1.00 1.0
    pastis 82800 20.0 1.00 0.000000 1.0 1.00 1.0 1.00 1.0
    aflplusplus 82800 20.0 0.40 0.502625 0.0 0.00 0.0 1.00 1.0
    aflplusplusplus 82800 20.0 0.20 0.410391 0.0 0.00 0.0 0.00 1.0
    hastefuzz 82800 20.0 0.00 0.000000 0.0 0.00 0.0 0.00 0.0
    libfuzzer 82800 20.0 0.25 0.444262 0.0 0.00 0.0 0.25 1.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 19756.80 176.947985 19393.0 19672.25 19750.5 19845.00 20059.0
    pastis 82800 20.0 19693.60 132.479353 19472.0 19604.50 19689.0 19747.00 20038.0
    aflsmart_plusplus 82800 20.0 19600.25 562.665636 17267.0 19634.75 19678.0 19756.50 20057.0
    aflpp_d82218 82800 20.0 19418.75 134.450881 19125.0 19337.25 19392.0 19492.50 19695.0
    aflppp_d82218 82800 20.0 19213.50 161.700666 18871.0 19149.75 19239.0 19285.25 19609.0
    aflrustrust 82800 20.0 18786.85 929.576829 16788.0 18892.50 19160.5 19313.25 19703.0
    aflplusplus 82800 20.0 19090.00 198.425115 18517.0 19027.50 19121.0 19222.50 19334.0
    aflplusplusplus 82800 20.0 18388.10 926.437142 16271.0 18282.25 18717.5 18917.25 19315.0
    hastefuzz 82800 20.0 18012.80 991.476124 16107.0 17872.25 18322.0 18702.75 18991.0
    honggfuzz 82800 20.0 17095.90 55.619856 16998.0 17067.50 17096.5 17124.75 17218.0
    libfuzzer 82800 20.0 15525.50 550.380586 14618.0 15296.25 15409.5 15719.75 17195.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

mbedtls_fuzz_dtlsclient_7c6b0e summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    aflplusplus 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    aflplusplusplus 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    aflrustrust 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    aflsmart_plusplus 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    hastefuzz 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    honggfuzz 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    libfuzzer 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    hastefuzz 82800 20.0 2635.20 9.168596 2618.0 2630.75 2637.0 2638.00 2662.0
    aflsmart_plusplus 82800 20.0 2843.00 476.554741 2517.0 2615.75 2625.5 2638.75 3789.0
    aflplusplusplus 82800 20.0 2614.45 18.503129 2562.0 2607.00 2613.5 2625.50 2645.0
    afl 82800 20.0 2608.95 11.500458 2589.0 2602.75 2612.5 2616.50 2629.0
    honggfuzz 82800 20.0 2655.10 194.723258 2553.0 2582.00 2600.0 2611.00 3283.0
    aflrustrust 82800 20.0 2732.70 227.217054 2543.0 2566.25 2596.5 2915.25 3192.0
    libfuzzer 82800 20.0 2588.35 15.428187 2557.0 2580.25 2587.5 2592.25 2620.0
    aflplusplus 82800 20.0 2589.95 19.645476 2570.0 2573.75 2583.0 2605.00 2631.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

php_php-fuzz-parser_0dbedb summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 0.200000 0.410391 0.0 0.0 0.0 0.0 1.0
    aflplusplus 82800 20.0 0.100000 0.307794 0.0 0.0 0.0 0.0 1.0
    aflplusplusplus 82800 20.0 0.100000 0.307794 0.0 0.0 0.0 0.0 1.0
    aflpp_d82218 82800 20.0 0.000000 0.000000 0.0 0.0 0.0 0.0 0.0
    aflppp_d82218 82800 20.0 0.150000 0.366348 0.0 0.0 0.0 0.0 1.0
    aflrustrust 82800 20.0 0.000000 0.000000 0.0 0.0 0.0 0.0 0.0
    aflsmart_plusplus 82800 20.0 0.350000 0.489360 0.0 0.0 0.0 1.0 1.0
    hastefuzz 82800 20.0 0.000000 0.000000 0.0 0.0 0.0 0.0 0.0
    honggfuzz 82800 20.0 0.150000 0.489360 0.0 0.0 0.0 0.0 2.0
    libfuzzer 82800 20.0 0.000000 0.000000 0.0 0.0 0.0 0.0 0.0
    pastis 82800 17.0 0.176471 0.392953 0.0 0.0 0.0 0.0 1.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    honggfuzz 82800 20.0 17082.550000 114.516593 16835.0 17003.75 17088.0 17156.75 17293.0
    pastis 82800 17.0 16968.764706 65.323741 16820.0 16958.00 16965.0 16983.00 17131.0
    aflpp_d82218 82800 20.0 16771.450000 69.338586 16656.0 16741.50 16761.0 16780.25 17006.0
    aflppp_d82218 82800 20.0 16777.650000 109.817684 16554.0 16726.00 16761.0 16846.75 17036.0
    aflplusplus 82800 20.0 16767.150000 94.605149 16672.0 16704.50 16729.5 16787.75 16990.0
    aflplusplusplus 82800 20.0 16742.550000 98.482953 16606.0 16691.50 16715.5 16774.75 16989.0
    aflsmart_plusplus 82800 20.0 16689.600000 47.847785 16645.0 16661.50 16674.5 16688.25 16812.0
    afl 82800 20.0 16664.800000 23.460157 16622.0 16651.75 16663.5 16678.25 16704.0
    hastefuzz 82800 20.0 16580.100000 29.875003 16518.0 16556.75 16581.0 16604.25 16629.0
    aflrustrust 82800 20.0 16481.600000 43.711254 16438.0 16461.50 16468.0 16492.50 16644.0
    libfuzzer 82800 20.0 16456.350000 59.187725 16374.0 16423.25 16455.0 16475.25 16597.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

systemd_fuzz-network-parser_288baf summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 0.00 0.000000 0.0 0.0 0.0 0.0 0.0
    aflplusplus 82800 20.0 0.00 0.000000 0.0 0.0 0.0 0.0 0.0
    aflplusplusplus 82800 20.0 0.00 0.000000 0.0 0.0 0.0 0.0 0.0
    aflpp_d82218 82800 20.0 0.00 0.000000 0.0 0.0 0.0 0.0 0.0
    aflppp_d82218 82800 20.0 0.00 0.000000 0.0 0.0 0.0 0.0 0.0
    aflrustrust 82800 20.0 0.15 0.670820 0.0 0.0 0.0 0.0 3.0
    aflsmart_plusplus 82800 20.0 0.00 0.000000 0.0 0.0 0.0 0.0 0.0
    hastefuzz 82800 20.0 0.00 0.000000 0.0 0.0 0.0 0.0 0.0
    honggfuzz 82800 20.0 0.45 1.099043 0.0 0.0 0.0 0.0 3.0
    libfuzzer 82800 20.0 0.00 0.000000 0.0 0.0 0.0 0.0 0.0
    pastis 82800 19.0 0.00 0.000000 0.0 0.0 0.0 0.0 0.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflrustrust 82800 20.0 3569.650000 26.764617 3526.0 3545.00 3577.0 3588.25 3621.0
    aflppp_d82218 82800 20.0 3520.900000 43.503055 3386.0 3517.75 3534.5 3541.50 3579.0
    aflpp_d82218 82800 20.0 3396.700000 27.921130 3347.0 3378.75 3404.0 3417.50 3438.0
    afl 82800 20.0 3400.050000 22.129939 3341.0 3388.75 3399.0 3410.25 3447.0
    aflsmart_plusplus 82800 20.0 3383.250000 22.961925 3332.0 3370.50 3387.0 3398.75 3414.0
    aflplusplusplus 82800 20.0 3274.850000 82.052150 3105.0 3233.50 3276.0 3337.75 3389.0
    libfuzzer 82800 20.0 2885.100000 683.519984 1880.0 2039.25 3260.0 3424.00 3524.0
    aflplusplus 82800 20.0 3131.700000 29.679162 3083.0 3111.00 3126.5 3148.00 3202.0
    hastefuzz 82800 20.0 3138.100000 40.270010 3078.0 3109.50 3125.5 3160.75 3227.0
    honggfuzz 82800 20.0 2771.350000 259.213847 2562.0 2650.25 2701.5 2746.25 3505.0
    pastis 82800 19.0 1618.578947 4.970086 1609.0 1613.00 1622.0 1622.00 1623.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

experiment data

You can download the raw data for this report here.

Check out the documentation on how to create customized reports using this data. Also see some example Colab notebooks for doing custom analysis on the data here.

Experiment Description:

(None,)