FuzzBench: 2023-05-08-test-new-aflpp report

experiment summary

We show two different aggregate (cross-benchmark) rankings of fuzzers. The first is based on the average of per-benchmarks scores, where the score represents the percentage of the highest reached median code-coverage on a given benchmark (higher value is better). The second ranking shows the average rank of fuzzers, after we rank them on each benchmark according to their median reached code-covereges (lower value is better).
By avg. score
average normalized score
fuzzer
aflplusplusplus 96.68
aflplusplus 96.05
aflpp_d82218 95.47
aflppp_d82218 95.03
hastefuzz 94.99
libafl 61.93
aflrustrust 61.79
eclipser 56.06
mopt 53.64
aflsmart 53.05
aflfast 52.06
fairfuzz 50.50
pastis 12.11
centipede 5.50
libafl_libfuzzer 2.88
afl 2.43
libfuzzer 2.41
honggfuzz 2.41
By avg. rank
average rank
fuzzer
aflpp_d82218 2.79
aflppp_d82218 3.36
hastefuzz 3.85
aflplusplus 4.18
aflplusplusplus 4.30
libafl 5.76
aflrustrust 5.85
eclipser 7.00
aflsmart 7.27
mopt 7.79
aflfast 8.79
fairfuzz 9.00
pastis 9.91
centipede 10.30
libafl_libfuzzer 10.45
afl 10.45
libfuzzer 10.52
honggfuzz 10.55
  • Critical difference diagram
    The diagram visualizes the average rank of fuzzers (second ranking above) while showing the significance of the differences as well. What is considered a "critical difference" (CD) is based on the Friedman/Nemenyi post-hoc test. See more in the documentation.
    Note: If a fuzzer does not support all benchmarks, its ranking as shown in this diagram can be lower than it should be. So please check the list of supported benchmarks for the fuzzer(s) of your interest. The list could be specified in the fuzzer's README.md like this.
  • Median relative code-coverages on each benchmark

    Note: The relative coverage summary table shows the median relative performance of each fuzzer to the experiment maximum. Thus the highest relative performance may not be 100%.
    trial_relative_coverage = trial_coverage / experiment_max_coverage

      pastis aflpp_d82218 aflppp_d82218 hastefuzz libafl_libfuzzer libafl aflrustrust aflplusplusplus aflplusplus eclipser mopt aflsmart aflfast fairfuzz honggfuzz libfuzzer afl centipede
    FuzzerMedian 99.00 99.00 99.00 98.50 94.00 97.00 96.00 98.00 98.00 94.00 93.50 94.50 93.00 85.50 70.00 70.00 70.00 68.00
    FuzzerMean 97.75 95.12 94.75 94.44 94.00 93.95 93.71 92.97 92.36 84.95 77.86 77.14 75.73 73.41 70.00 70.00 70.00 56.67
    arduinojson_json_fuzzer 99.00 99.00 99.00 99.00 nan nan nan 99.00 99.00 nan nan nan nan nan nan nan nan nan
    bloaty_fuzz_target nan 97.00 97.00 75.00 nan 98.00 92.00 75.00 98.00 94.00 96.00 95.00 93.00 80.00 nan nan nan nan
    botan_tls_server nan 51.00 50.00 64.00 nan nan nan 64.00 65.00 nan nan nan nan nan nan nan nan nan
    curl_curl_fuzzer_http nan 98.00 98.00 98.00 nan 97.00 97.00 98.00 97.00 95.00 94.00 94.00 93.00 84.00 nan nan nan nan
    double-conversion_string_to_double_fuzzer nan 99.00 99.00 99.00 nan nan 98.00 99.00 99.00 nan nan nan nan nan nan nan nan nan
    draco_draco_pc_decoder_fuzzer nan 90.00 86.00 89.00 nan nan nan 87.00 89.00 nan nan nan nan nan nan nan nan nan
    firestore_firestore_serializer_fuzzer nan 100.00 100.00 100.00 nan nan nan 100.00 100.00 nan nan nan nan nan nan nan nan nan
    fmt_chrono-duration-fuzzer nan 99.00 99.00 99.00 nan nan nan 99.00 99.00 nan nan nan nan nan nan nan nan nan
    freetype2_ftfuzzer nan 95.00 96.00 93.00 nan 90.00 84.00 96.00 82.00 74.00 67.00 67.00 64.00 63.00 nan nan nan nan
    guetzli_guetzli_fuzzer nan 99.00 99.00 99.00 nan nan nan 99.00 99.00 nan nan nan nan nan nan nan nan nan
    harfbuzz_hb-shape-fuzzer nan 98.00 98.00 98.00 nan 99.00 99.00 98.00 98.00 97.00 97.00 97.00 95.00 87.00 nan nan nan nan
    icu_unicode_string_codepage_create_fuzzer 100.00 100.00 100.00 99.00 nan nan nan 99.00 99.00 nan nan nan nan nan nan nan nan nan
    jansson_json_load_dump_fuzzer nan 99.00 99.00 99.00 nan nan nan 99.00 99.00 nan nan nan nan nan nan nan nan nan
    jsoncpp_jsoncpp_fuzzer nan 99.00 99.00 99.00 nan 98.00 98.00 99.00 99.00 98.00 98.00 98.00 98.00 98.00 nan nan nan nan
    lcms_cms_transform_fuzzer nan 87.00 78.00 89.00 nan 90.00 90.00 77.00 71.00 78.00 62.00 40.00 28.00 50.00 nan nan nan nan
    libaom_av1_dec_fuzzer nan 99.00 99.00 96.00 nan nan nan 94.00 96.00 nan nan nan nan nan nan nan nan nan
    libcoap_pdu_parse_fuzzer nan 99.00 99.00 99.00 nan nan nan 99.00 99.00 nan nan nan nan nan nan nan nan nan
    libhevc_hevc_dec_fuzzer 99.00 99.00 99.00 99.00 nan nan nan 99.00 99.00 0.00 0.00 0.00 0.00 0.00 nan nan nan 5.00
    libjpeg-turbo_libjpeg_turbo_fuzzer nan 99.00 99.00 99.00 nan 99.00 99.00 99.00 99.00 nan 99.00 99.00 99.00 99.00 nan nan nan nan
    libpcap_fuzz_both nan 93.00 94.00 94.00 nan 86.00 88.00 93.00 90.00 81.00 1.00 1.00 1.00 1.00 nan nan nan nan
    libpng_libpng_read_fuzzer nan 95.00 95.00 95.00 nan 95.00 94.00 95.00 95.00 95.00 95.00 95.00 93.00 95.00 nan nan nan nan
    libxml2_xml nan 99.00 99.00 96.00 nan 98.00 98.00 96.00 97.00 97.00 96.00 97.00 96.00 89.00 nan nan nan nan
    libxslt_xpath nan 99.00 98.00 98.00 nan 96.00 96.00 98.00 99.00 96.00 93.00 96.00 94.00 97.00 nan nan nan nan
    mbedtls_fuzz_dtlsclient nan nan nan nan nan 75.00 87.00 72.00 71.00 70.00 69.00 70.00 68.00 71.00 70.00 70.00 70.00 68.00
    openh264_decoder_fuzzer nan 99.00 99.00 99.00 nan 99.00 99.00 99.00 99.00 99.00 99.00 99.00 99.00 90.00 nan nan nan nan
    openssl_x509 nan 99.00 99.00 99.00 nan 99.00 99.00 99.00 99.00 99.00 99.00 99.00 99.00 99.00 nan nan nan nan
    openthread_ot-ip6-send-fuzzer nan 76.00 77.00 82.00 nan 82.00 89.00 77.00 76.00 76.00 73.00 73.00 72.00 69.00 nan nan nan nan
    proj4_proj_crs_to_crs_fuzzer 93.00 91.00 91.00 94.00 nan 93.00 nan 91.00 71.00 63.00 10.00 10.00 10.00 10.00 nan nan nan nan
    re2_fuzzer nan 99.00 99.00 99.00 nan 98.00 98.00 99.00 99.00 99.00 99.00 99.00 99.00 99.00 nan nan nan nan
    sqlite3_ossfuzz nan 98.00 98.00 84.00 nan 90.00 79.00 80.00 85.00 93.00 92.00 93.00 90.00 65.00 nan nan nan nan
    stb_stbi_read_fuzzer nan 92.00 92.00 92.00 nan 95.00 91.00 92.00 89.00 91.00 86.00 87.00 86.00 86.00 nan nan nan nan
    systemd_fuzz-link-parser nan 99.00 99.00 99.00 94.00 98.00 94.00 99.00 93.00 91.00 90.00 90.00 91.00 85.00 nan nan nan 97.00
    vorbis_decode_fuzzer nan 99.00 99.00 99.00 nan 98.00 99.00 99.00 99.00 98.00 98.00 98.00 98.00 98.00 nan nan nan nan
    • Fuzzers are sorted by "FuzzerMean" (average median relative coverage), highest on the left.
    • Green background = highest relative median coverage.
    • Blue gradient background = greater than 95% relative median coverage.

arduinojson_json_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus 82800 20.0 401.600000 0.502625 401.0 401.0 402.0 402.0 402.0
    aflpp_d82218 82800 20.0 401.700000 0.470162 401.0 401.0 402.0 402.0 402.0
    aflppp_d82218 82800 20.0 401.800000 0.410391 401.0 402.0 402.0 402.0 402.0
    hastefuzz 82800 20.0 401.600000 0.502625 401.0 401.0 402.0 402.0 402.0
    pastis 82800 17.0 402.058824 0.747545 401.0 402.0 402.0 403.0 403.0
    aflplusplusplus 82800 20.0 401.500000 0.512989 401.0 401.0 401.5 402.0 402.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

bloaty_fuzz_target summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus 82800 20.0 6357.55 74.439290 6162.0 6341.00 6374.5 6402.00 6456.0
    libafl 82800 20.0 6353.80 71.997515 6140.0 6344.75 6371.5 6388.00 6442.0
    aflpp_d82218 82800 20.0 6338.30 73.219281 6196.0 6299.50 6342.0 6402.75 6447.0
    aflppp_d82218 82800 20.0 6314.85 116.055465 6107.0 6263.75 6303.0 6396.25 6485.0
    mopt 82800 20.0 6236.05 71.516266 6071.0 6226.50 6243.5 6268.50 6354.0
    aflsmart 82800 20.0 6198.45 127.984981 5937.0 6102.25 6205.0 6295.25 6394.0
    eclipser 82800 20.0 6110.20 114.487232 5900.0 6066.75 6107.0 6144.50 6360.0
    aflfast 82800 20.0 6080.10 119.230824 5858.0 6033.50 6059.0 6136.50 6350.0
    aflrustrust 82800 20.0 5980.95 110.845305 5754.0 5900.25 5986.5 6045.75 6157.0
    fairfuzz 82800 20.0 5217.75 121.813868 5060.0 5145.00 5194.5 5272.75 5570.0
    aflplusplusplus 82800 20.0 4914.90 78.681169 4783.0 4857.25 4922.5 4966.00 5086.0
    hastefuzz 82800 20.0 4872.50 115.648970 4617.0 4812.25 4871.5 4925.50 5122.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

botan_tls_server summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus 82800 20.0 1887.45 433.127607 1252.0 1626.00 1960.0 2215.00 2585.0
    aflplusplusplus 82800 20.0 1931.60 484.435798 1183.0 1707.25 1925.0 2364.75 2980.0
    hastefuzz 82800 20.0 1842.75 412.196794 1184.0 1530.25 1910.0 2123.75 2384.0
    aflpp_d82218 82800 20.0 1553.55 330.905532 1176.0 1256.00 1520.0 1869.50 1988.0
    aflppp_d82218 82800 20.0 1722.35 533.646273 1176.0 1257.00 1512.5 2307.25 2721.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

curl_curl_fuzzer_http summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflpp_d82218 82800 20.0 10846.40 75.292692 10699.0 10801.00 10828.0 10871.25 10991.0
    hastefuzz 82800 20.0 10781.05 56.700019 10652.0 10755.75 10787.5 10807.25 10889.0
    aflplusplusplus 82800 20.0 10799.90 56.766280 10713.0 10766.75 10783.5 10846.75 10913.0
    aflppp_d82218 82800 20.0 10782.40 56.524238 10714.0 10734.00 10771.5 10814.25 10898.0
    aflplusplus 82800 20.0 10705.40 42.374769 10626.0 10675.00 10712.5 10735.50 10772.0
    libafl 82800 20.0 10690.00 54.699177 10599.0 10655.25 10684.5 10723.75 10802.0
    aflrustrust 82800 20.0 10658.10 57.697852 10561.0 10612.00 10666.5 10700.50 10753.0
    eclipser 82800 20.0 10463.90 37.160959 10415.0 10437.00 10454.0 10491.00 10551.0
    aflsmart 82800 20.0 10456.75 63.366831 10389.0 10399.00 10436.5 10486.75 10575.0
    mopt 82800 20.0 10338.90 115.764234 10036.0 10331.75 10376.0 10408.50 10441.0
    aflfast 82800 20.0 10272.85 106.610592 9985.0 10221.25 10305.5 10346.25 10406.0
    fairfuzz 82800 20.0 9200.80 420.958005 7990.0 9035.75 9266.0 9472.50 9906.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

double-conversion_string_to_double_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
error
The following fuzzers do not have enough samples: aflrustrust.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus 82800 20.0 507.75 1.831738 504.0 506.00 508.0 509.00 510.0
    hastefuzz 82800 20.0 507.35 1.598519 504.0 506.00 508.0 508.00 510.0
    aflppp_d82218 82800 20.0 506.80 1.794729 503.0 506.00 507.0 508.00 510.0
    aflpp_d82218 82800 20.0 506.50 2.064742 504.0 504.75 506.5 508.00 510.0
    aflplusplusplus 82800 20.0 506.10 1.832456 503.0 504.75 506.0 507.00 510.0
    aflrustrust 82800 2.0 503.50 0.707107 503.0 503.25 503.5 503.75 504.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

draco_draco_pc_decoder_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflpp_d82218 81900 20.0 1838.15 90.499360 1699.0 1805.25 1822.5 1847.00 2022.0
    aflplusplus 81900 20.0 1827.80 93.341817 1698.0 1754.50 1815.5 1913.50 1996.0
    hastefuzz 81900 20.0 1793.10 70.810273 1649.0 1772.00 1804.0 1815.25 1976.0
    aflplusplusplus 81900 20.0 1790.85 57.708227 1741.0 1749.75 1769.0 1811.25 1963.0
    aflppp_d82218 81900 20.0 1775.90 94.638870 1620.0 1726.00 1759.5 1826.75 2024.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

firestore_firestore_serializer_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus 82800 20.0 287.0 0.0 287.0 287.0 287.0 287.0 287.0
    aflplusplusplus 82800 20.0 287.0 0.0 287.0 287.0 287.0 287.0 287.0
    aflpp_d82218 82800 20.0 287.0 0.0 287.0 287.0 287.0 287.0 287.0
    aflppp_d82218 82800 20.0 287.0 0.0 287.0 287.0 287.0 287.0 287.0
    hastefuzz 82800 20.0 287.0 0.0 287.0 287.0 287.0 287.0 287.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

fmt_chrono-duration-fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflppp_d82218 82800 20.0 1095.85 2.390221 1089.0 1096.00 1096.0 1097.00 1099.0
    aflpp_d82218 82800 20.0 1095.70 2.296450 1091.0 1094.75 1095.5 1097.00 1099.0
    hastefuzz 82800 20.0 1095.05 3.103055 1089.0 1093.00 1095.5 1097.25 1100.0
    aflplusplusplus 82800 20.0 1094.40 3.015748 1089.0 1092.50 1095.0 1097.00 1098.0
    aflplusplus 82800 20.0 1089.80 3.607011 1081.0 1086.75 1091.0 1093.00 1095.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

freetype2_ftfuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplusplus 82800 20.0 11894.00 221.972497 11328.0 11820.50 11929.0 12044.75 12210.0
    aflppp_d82218 82800 20.0 11847.20 238.264470 10973.0 11807.50 11877.0 11952.25 12165.0
    aflpp_d82218 82800 20.0 11730.85 325.371897 10996.0 11610.75 11778.0 11918.25 12238.0
    hastefuzz 82800 20.0 11552.25 405.614398 10431.0 11424.25 11611.5 11841.50 12141.0
    libafl 82800 20.0 11258.70 576.622023 10133.0 10872.00 11183.5 11721.75 12353.0
    aflrustrust 82800 20.0 10425.20 131.442042 10211.0 10344.25 10394.0 10508.25 10715.0
    aflplusplus 82800 20.0 10217.50 210.242341 9888.0 10073.75 10211.5 10340.75 10781.0
    eclipser 82800 20.0 9242.10 100.454703 9028.0 9198.75 9263.0 9316.75 9423.0
    mopt 82800 20.0 8332.40 177.225815 7888.0 8320.75 8377.5 8442.00 8504.0
    aflsmart 82800 20.0 8244.00 202.383690 7800.0 8230.50 8304.5 8374.50 8524.0
    aflfast 82800 20.0 7976.35 222.903159 7640.0 7820.50 7916.0 8204.75 8311.0
    fairfuzz 82800 20.0 7832.95 112.700570 7683.0 7768.00 7816.0 7859.75 8213.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

guetzli_guetzli_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflpp_d82218 82800 20.0 1498.45 11.440809 1451.0 1499.0 1501.0 1502.25 1506.0
    hastefuzz 82800 20.0 1497.65 7.506313 1480.0 1492.0 1501.0 1502.25 1506.0
    aflplusplus 82800 20.0 1498.30 5.391026 1484.0 1495.0 1498.5 1502.25 1506.0
    aflppp_d82218 82800 20.0 1495.00 7.766324 1480.0 1487.0 1498.0 1502.00 1504.0
    aflplusplusplus 82800 20.0 1494.25 8.521644 1483.0 1486.0 1493.5 1503.00 1505.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

harfbuzz_hb-shape-fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflrustrust 82800 20.0 10972.40 239.429673 10118.0 10995.50 11048.5 11089.75 11135.0
    libafl 82800 20.0 11027.85 34.580608 10961.0 11006.50 11026.5 11049.75 11087.0
    aflplusplus 82800 20.0 10978.60 65.055766 10842.0 10934.75 10994.5 11021.50 11071.0
    aflpp_d82218 82800 20.0 10968.70 58.777099 10797.0 10933.50 10991.5 11003.50 11054.0
    aflplusplusplus 82800 20.0 10975.20 40.145263 10905.0 10956.75 10975.5 10999.25 11045.0
    aflppp_d82218 82800 20.0 10972.65 43.646034 10858.0 10944.75 10972.0 11001.75 11048.0
    hastefuzz 82800 20.0 10961.30 37.032134 10893.0 10932.75 10960.5 10991.25 11032.0
    aflsmart 82800 20.0 10815.45 39.488139 10703.0 10800.25 10821.0 10841.25 10879.0
    eclipser 82800 20.0 10820.15 43.008292 10754.0 10778.50 10819.5 10861.50 10885.0
    mopt 82800 20.0 10796.70 40.874777 10699.0 10777.25 10801.5 10821.00 10864.0
    aflfast 82800 20.0 10670.70 51.265152 10567.0 10642.75 10669.0 10702.75 10770.0
    fairfuzz 82800 20.0 9728.25 323.424257 9130.0 9481.50 9724.0 9986.00 10218.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

icu_unicode_string_codepage_create_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
error
The following fuzzers do not have enough samples: pastis.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflpp_d82218 82800 20.0 1338.750 8.662533 1302.0 1340.0 1341.0 1341.0 1341.0
    aflppp_d82218 82800 20.0 1340.450 0.759155 1339.0 1340.0 1341.0 1341.0 1341.0
    pastis 82800 8.0 1340.875 0.353553 1340.0 1341.0 1341.0 1341.0 1341.0
    aflplusplus 82800 20.0 1339.850 0.745160 1339.0 1339.0 1340.0 1340.0 1341.0
    aflplusplusplus 82800 20.0 1340.000 0.561951 1339.0 1340.0 1340.0 1340.0 1341.0
    hastefuzz 82800 20.0 1340.400 0.598243 1339.0 1340.0 1340.0 1341.0 1341.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

jansson_json_load_dump_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflpp_d82218 82800 20.0 768.45 1.669384 765.0 767.0 769.0 770.00 771.0
    hastefuzz 82800 20.0 768.60 1.095445 766.0 768.0 768.5 769.25 770.0
    aflplusplus 82800 20.0 767.80 1.472556 765.0 767.0 768.0 769.00 770.0
    aflplusplusplus 82800 20.0 768.10 1.293709 766.0 767.0 768.0 769.00 770.0
    aflppp_d82218 82800 20.0 767.70 1.490320 765.0 767.0 768.0 768.00 771.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

jsoncpp_jsoncpp_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus 82800 20.0 519.80 0.410391 519.0 520.00 520.0 520.00 520.0
    aflplusplusplus 82800 20.0 520.00 0.000000 520.0 520.00 520.0 520.00 520.0
    aflpp_d82218 82800 20.0 519.75 0.444262 519.0 519.75 520.0 520.00 520.0
    aflppp_d82218 82800 20.0 519.90 0.307794 519.0 520.00 520.0 520.00 520.0
    hastefuzz 82800 20.0 519.70 0.470162 519.0 519.00 520.0 520.00 520.0
    eclipser 82800 20.0 518.90 1.252366 516.0 518.00 519.5 520.00 520.0
    aflrustrust 82800 20.0 519.00 0.000000 519.0 519.00 519.0 519.00 519.0
    mopt 82800 20.0 517.20 3.955010 505.0 518.00 518.0 519.00 520.0
    aflsmart 82800 20.0 517.90 1.165287 516.0 517.00 517.5 519.00 520.0
    aflfast 82800 20.0 513.90 10.622221 478.0 517.00 517.0 519.00 519.0
    fairfuzz 82800 20.0 517.45 0.998683 516.0 517.00 517.0 517.25 520.0
    libafl 82800 20.0 517.25 0.850696 516.0 517.00 517.0 517.00 519.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

lcms_cms_transform_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflrustrust 82800 20.0 2091.50 88.865659 1903.0 2027.50 2074.0 2159.75 2286.0
    libafl 82800 20.0 2076.20 108.050476 1894.0 1999.00 2071.0 2161.25 2287.0
    hastefuzz 82800 20.0 1968.15 171.629735 1568.0 1933.50 2044.5 2063.50 2138.0
    aflpp_d82218 82800 20.0 1902.65 238.724742 1299.0 1767.25 2005.5 2062.75 2132.0
    aflppp_d82218 82800 20.0 1658.60 428.844273 696.0 1719.00 1797.5 1896.25 2063.0
    eclipser 82800 20.0 1736.00 135.909025 1522.0 1573.00 1789.0 1838.50 1904.0
    aflplusplusplus 82800 20.0 1603.20 441.165514 697.0 1572.25 1765.5 1867.75 2120.0
    aflplusplus 82800 20.0 1615.00 130.427233 1411.0 1507.50 1638.5 1746.75 1779.0
    mopt 82800 20.0 1230.00 430.050425 651.0 845.25 1439.5 1604.00 1765.0
    fairfuzz 82800 20.0 1181.85 428.681488 652.0 790.25 1148.5 1575.75 1919.0
    aflsmart 82800 20.0 1123.25 441.957235 649.0 673.75 918.0 1575.50 1795.0
    aflfast 82800 20.0 635.60 19.277475 569.0 637.75 642.5 646.00 657.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

libaom_av1_dec_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflpp_d82218 82800 20.0 11408.35 42.927448 11298.0 11393.25 11412.0 11436.50 11485.0
    aflppp_d82218 82800 20.0 11396.50 54.152319 11237.0 11387.75 11410.0 11433.00 11462.0
    aflplusplus 82800 20.0 11048.45 295.341457 10282.0 10997.75 11104.5 11242.25 11352.0
    hastefuzz 82800 20.0 11082.05 226.424026 10388.0 10950.25 11103.5 11265.50 11365.0
    aflplusplusplus 82800 20.0 10892.95 128.162921 10669.0 10831.50 10866.0 10945.00 11210.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

libcoap_pdu_parse_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflpp_d82218 82800 20.0 817.35 1.268028 815.0 816.0 817.5 818.0 820.0
    aflplusplus 82800 20.0 817.35 1.268028 815.0 817.0 817.0 818.0 821.0
    aflplusplusplus 82800 20.0 817.35 1.089423 816.0 817.0 817.0 818.0 820.0
    aflppp_d82218 82800 20.0 810.50 21.239239 748.0 816.0 817.0 818.0 820.0
    hastefuzz 82800 20.0 817.10 1.518309 814.0 817.0 817.0 818.0 819.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

libhevc_hevc_dec_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
error
The following fuzzers do not have enough samples: pastis.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflppp_d82218 82800 20.0 10364.30 11.544422 10329.0 10358.50 10365.5 10374.00 10378.0
    aflpp_d82218 82800 20.0 10364.90 8.347076 10349.0 10358.75 10365.0 10371.50 10377.0
    aflplusplus 82800 20.0 10332.70 47.309841 10219.0 10331.50 10355.0 10360.00 10375.0
    aflplusplusplus 82800 20.0 10348.30 19.134152 10292.0 10348.00 10352.0 10357.50 10369.0
    hastefuzz 82800 20.0 10340.35 26.691759 10269.0 10338.25 10351.0 10356.00 10363.0
    pastis 82800 2.0 10345.50 9.192388 10339.0 10342.25 10345.5 10348.75 10352.0
    centipede 82800 20.0 1260.30 1723.585892 555.0 555.00 557.0 557.00 5651.0
    aflfast 82800 20.0 0.00 0.000000 0.0 0.00 0.0 0.00 0.0
    aflsmart 82800 20.0 0.00 0.000000 0.0 0.00 0.0 0.00 0.0
    eclipser 82800 17.0 0.00 0.000000 0.0 0.00 0.0 0.00 0.0
    fairfuzz 82800 20.0 0.00 0.000000 0.0 0.00 0.0 0.00 0.0
    mopt 82800 20.0 0.00 0.000000 0.0 0.00 0.0 0.00 0.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

libjpeg-turbo_libjpeg_turbo_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus 82800 20.0 2549.10 1.997367 2546.0 2547.00 2549.5 2551.00 2552.0
    aflpp_d82218 82800 20.0 2548.80 1.908430 2546.0 2547.00 2548.5 2551.00 2552.0
    aflppp_d82218 82800 20.0 2548.50 2.212405 2546.0 2547.00 2547.5 2551.00 2552.0
    hastefuzz 82800 20.0 2548.45 2.089447 2546.0 2547.00 2547.5 2550.25 2552.0
    aflplusplusplus 82800 20.0 2547.95 2.038446 2546.0 2546.75 2547.0 2548.75 2552.0
    aflrustrust 82800 20.0 2545.80 2.238420 2543.0 2543.00 2547.0 2548.00 2549.0
    aflsmart 82800 20.0 2545.35 1.980829 2543.0 2544.00 2545.0 2546.00 2551.0
    mopt 82800 20.0 2544.45 1.099043 2543.0 2544.00 2544.0 2545.00 2547.0
    aflfast 82800 20.0 2543.90 1.483240 2541.0 2543.00 2543.5 2545.00 2548.0
    libafl 82800 20.0 2543.45 0.604805 2543.0 2543.00 2543.0 2544.00 2545.0
    fairfuzz 82800 20.0 2539.95 4.795557 2533.0 2536.00 2542.0 2542.25 2550.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

libpcap_fuzz_both summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflppp_d82218 82800 20.0 2949.150000 110.544240 2750.0 2888.00 2964.0 3024.00 3139.0
    hastefuzz 82800 19.0 2947.631579 113.117054 2771.0 2865.00 2961.0 3026.50 3147.0
    aflplusplusplus 82800 20.0 2918.300000 104.882440 2707.0 2848.00 2945.5 2997.50 3081.0
    aflpp_d82218 82800 20.0 2951.500000 109.146690 2692.0 2890.25 2943.0 3027.75 3116.0
    aflplusplus 82800 20.0 2823.700000 84.868381 2600.0 2779.75 2835.0 2879.25 2944.0
    aflrustrust 82800 20.0 2761.800000 126.655189 2499.0 2670.25 2783.5 2862.50 2932.0
    libafl 82800 20.0 2734.200000 108.331655 2527.0 2678.00 2715.0 2773.50 2989.0
    eclipser 82800 20.0 2506.050000 205.275547 1990.0 2411.75 2553.5 2590.75 2872.0
    aflfast 82800 20.0 40.300000 4.231461 34.0 34.00 43.0 43.00 43.0
    fairfuzz 82800 20.0 38.850000 4.404244 33.0 33.00 42.0 42.00 42.0
    mopt 82800 20.0 37.400000 4.357691 34.0 34.00 34.0 43.00 43.0
    aflsmart 82800 20.0 34.800000 3.693522 33.0 33.00 33.0 33.00 42.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

libpng_libpng_read_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflpp_d82218 82800 20.0 2009.60 18.285456 2003.0 2004.75 2005.5 2007.00 2087.0
    aflplusplusplus 82800 20.0 2013.30 26.630216 2002.0 2003.00 2005.0 2006.25 2091.0
    aflppp_d82218 82800 20.0 2007.90 19.306121 1992.0 2003.00 2004.5 2006.25 2088.0
    hastefuzz 82800 20.0 2011.95 25.485755 1997.0 2002.00 2004.5 2006.00 2087.0
    aflplusplus 82800 20.0 2000.15 1.725200 1997.0 1999.00 2000.0 2002.00 2003.0
    libafl 82800 20.0 1992.85 27.054088 1949.0 1976.75 1997.5 2002.00 2088.0
    aflsmart 82800 20.0 1995.40 5.490662 1974.0 1994.00 1996.0 1998.25 2000.0
    eclipser 82800 20.0 1990.45 11.250614 1972.0 1978.00 1994.0 1998.25 2003.0
    mopt 82800 20.0 1978.85 26.776413 1911.0 1975.00 1990.0 1993.50 2008.0
    fairfuzz 82800 20.0 1984.10 13.814180 1945.0 1974.75 1988.5 1995.00 2000.0
    aflrustrust 82800 20.0 1987.75 9.797287 1978.0 1980.75 1983.0 1998.75 2005.0
    aflfast 82800 20.0 1967.60 20.301413 1943.0 1948.25 1964.0 1989.50 1997.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

libxml2_xml summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflppp_d82218 82800 20.0 15732.95 38.738632 15641.0 15710.50 15731.5 15750.25 15816.0
    aflpp_d82218 82800 20.0 15719.50 45.251926 15638.0 15690.25 15712.0 15758.75 15794.0
    libafl 82800 20.0 15569.30 37.629915 15515.0 15539.00 15570.5 15584.50 15652.0
    aflrustrust 82800 20.0 15552.80 37.318826 15478.0 15524.75 15558.5 15584.25 15607.0
    eclipser 82800 20.0 15423.45 79.669366 15257.0 15387.50 15423.5 15463.75 15601.0
    aflsmart 82800 20.0 15366.80 68.816767 15248.0 15322.50 15366.5 15410.75 15497.0
    aflplusplus 82800 20.0 15348.85 92.847512 15186.0 15291.00 15346.0 15404.00 15534.0
    mopt 82800 20.0 15318.60 62.704319 15155.0 15285.50 15326.0 15366.25 15397.0
    aflfast 82800 20.0 15320.65 78.755635 15159.0 15271.75 15318.0 15356.00 15509.0
    aflplusplusplus 82800 20.0 15315.30 89.670214 15168.0 15251.75 15295.0 15353.75 15524.0
    hastefuzz 82800 20.0 15261.30 101.921074 15071.0 15178.00 15277.5 15343.00 15409.0
    fairfuzz 82800 20.0 14146.65 259.273738 13616.0 14060.75 14144.0 14227.25 14834.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

libxslt_xpath summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflpp_d82218 82800 20.0 11143.90 76.912733 10969.0 11104.75 11167.0 11174.00 11268.0
    aflplusplus 82800 20.0 11135.45 67.758453 11024.0 11077.50 11159.5 11180.50 11228.0
    aflppp_d82218 82800 20.0 11101.75 75.929243 10947.0 11030.50 11133.5 11162.75 11210.0
    aflplusplusplus 82800 20.0 11110.70 78.062795 10989.0 11041.50 11098.0 11177.75 11233.0
    hastefuzz 82800 20.0 11082.60 79.283236 10947.0 11011.50 11086.5 11160.50 11202.0
    fairfuzz 82800 20.0 10883.25 144.964197 10516.0 10827.75 10942.5 10959.25 11047.0
    aflsmart 82800 20.0 10894.55 51.799690 10814.0 10859.25 10901.0 10915.00 11026.0
    eclipser 82800 20.0 10895.65 65.938787 10729.0 10862.50 10897.5 10938.25 11010.0
    aflrustrust 82800 20.0 10879.95 41.625492 10796.0 10862.75 10887.0 10903.75 10943.0
    libafl 82800 20.0 10848.90 87.106650 10675.0 10784.00 10868.0 10895.00 11026.0
    aflfast 82800 20.0 10636.05 92.919022 10363.0 10591.25 10649.5 10692.75 10786.0
    mopt 82800 20.0 10571.50 126.561198 10325.0 10484.75 10580.0 10646.00 10781.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

mbedtls_fuzz_dtlsclient summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflrustrust 82800 20.0 3257.15 375.380687 2707.0 2761.50 3396.5 3522.75 3861.0
    libafl 82800 20.0 3039.50 343.799437 2692.0 2729.75 2921.5 3309.75 3671.0
    aflplusplusplus 82800 20.0 2800.35 39.219054 2747.0 2764.50 2805.0 2821.50 2888.0
    fairfuzz 82800 20.0 2820.70 203.472771 2730.0 2753.25 2779.0 2794.75 3676.0
    aflplusplus 82800 20.0 2795.50 131.149774 2727.0 2749.50 2766.0 2781.25 3339.0
    eclipser 82800 20.0 2774.45 217.678052 2702.0 2715.25 2726.0 2733.25 3697.0
    afl 82800 20.0 2772.80 250.391819 2693.0 2709.00 2720.0 2727.00 3835.0
    aflsmart 82800 20.0 2722.05 16.750412 2697.0 2712.50 2720.0 2728.25 2771.0
    libfuzzer 82800 20.0 2705.35 19.478125 2665.0 2695.75 2704.5 2716.50 2754.0
    honggfuzz 82800 20.0 2718.95 82.995545 2636.0 2682.25 2703.0 2723.75 3044.0
    mopt 82800 20.0 2644.60 88.620658 2444.0 2562.50 2688.0 2699.50 2756.0
    centipede 82800 20.0 2639.90 12.531644 2621.0 2630.50 2639.0 2644.50 2673.0
    aflfast 82800 20.0 2638.30 14.026666 2607.0 2630.75 2638.0 2643.50 2675.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

openh264_decoder_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflrustrust 82800 20.0 9537.95 24.095151 9462.0 9535.50 9542.5 9553.50 9568.0
    mopt 82800 20.0 9539.10 13.540971 9517.0 9525.75 9540.5 9548.25 9563.0
    libafl 82800 20.0 9536.95 11.315918 9498.0 9531.00 9539.0 9543.00 9550.0
    aflpp_d82218 82800 20.0 9540.25 15.092260 9500.0 9532.75 9536.5 9553.25 9565.0
    aflsmart 82800 20.0 9518.75 42.213586 9433.0 9511.75 9533.5 9543.00 9582.0
    aflppp_d82218 82800 20.0 9510.10 47.840635 9376.0 9499.75 9531.0 9539.00 9558.0
    eclipser 82800 20.0 9518.45 26.296838 9441.0 9515.25 9526.5 9533.75 9541.0
    aflfast 82800 20.0 9509.10 52.625689 9380.0 9493.00 9525.5 9551.00 9561.0
    hastefuzz 82800 20.0 9466.70 90.709542 9254.0 9398.25 9516.5 9525.25 9536.0
    aflplusplus 82800 20.0 9486.65 74.103253 9248.0 9468.00 9515.5 9530.50 9545.0
    aflplusplusplus 82800 20.0 9474.20 112.677743 9048.0 9491.25 9515.0 9525.50 9541.0
    fairfuzz 82800 20.0 8690.75 181.798001 8400.0 8579.25 8713.0 8780.50 9055.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

openssl_x509 summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    hastefuzz 82800 20.0 5831.15 5.091531 5817.0 5828.75 5834.0 5834.0 5835.0
    aflrustrust 82800 20.0 5829.05 4.925498 5815.0 5827.50 5831.0 5832.0 5834.0
    aflsmart 82800 20.0 5830.35 2.539685 5827.0 5829.00 5830.0 5831.0 5840.0
    eclipser 82800 20.0 5829.45 3.119970 5817.0 5829.00 5830.0 5831.0 5832.0
    aflplusplus 82800 20.0 5828.60 4.197744 5822.0 5825.00 5829.0 5832.0 5835.0
    mopt 82800 20.0 5824.90 6.290343 5810.0 5823.00 5828.0 5829.0 5829.0
    aflpp_d82218 82800 20.0 5824.45 8.016266 5805.0 5820.50 5827.0 5830.0 5833.0
    libafl 82800 20.0 5825.80 4.274773 5821.0 5822.50 5824.0 5829.5 5833.0
    aflppp_d82218 82800 20.0 5822.50 8.022993 5809.0 5816.25 5823.0 5829.0 5834.0
    fairfuzz 82800 20.0 5821.50 2.874571 5817.0 5819.00 5822.0 5823.0 5827.0
    aflplusplusplus 82800 20.0 5820.20 6.048923 5810.0 5816.00 5821.0 5825.0 5829.0
    aflfast 82800 20.0 5819.20 6.304551 5804.0 5816.75 5820.5 5824.0 5827.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

openthread_ot-ip6-send-fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflrustrust 82800 20.0 3339.45 270.625683 2933.0 3055.75 3539.5 3562.25 3592.0
    libafl 82800 20.0 3358.10 345.626342 3018.0 3047.50 3270.0 3570.25 3967.0
    hastefuzz 82800 20.0 3342.70 301.353106 3043.0 3066.00 3255.0 3589.50 3947.0
    aflplusplusplus 82800 20.0 3266.90 257.034689 3047.0 3060.00 3074.5 3565.25 3598.0
    aflppp_d82218 82800 20.0 3111.25 140.848883 2984.0 3045.75 3058.5 3072.25 3459.0
    aflpp_d82218 82800 20.0 3055.45 9.156390 3036.0 3049.00 3054.5 3061.50 3071.0
    aflplusplus 82800 20.0 3011.45 166.544913 2818.0 2832.00 3046.0 3063.25 3418.0
    eclipser 82800 20.0 3022.20 46.183387 2911.0 3010.75 3038.5 3046.75 3087.0
    mopt 82800 20.0 2945.55 143.939854 2807.0 2902.25 2914.0 2923.75 3490.0
    aflsmart 82800 20.0 2889.60 38.982317 2826.0 2836.50 2909.5 2913.25 2934.0
    aflfast 82800 20.0 2877.30 39.174508 2824.0 2830.75 2892.5 2908.75 2927.0
    fairfuzz 82800 20.0 2756.10 75.288847 2628.0 2724.25 2751.5 2788.50 2907.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

proj4_proj_crs_to_crs_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    hastefuzz 82800 20.0 7287.850000 195.196062 6964.0 7113.50 7299.0 7424.00 7685.0
    pastis 82800 17.0 7324.882353 203.319909 6988.0 7231.00 7275.0 7424.00 7683.0
    libafl 82800 20.0 7216.700000 122.786550 6946.0 7142.25 7209.5 7274.25 7435.0
    aflplusplusplus 82800 20.0 7092.300000 137.201389 6719.0 7004.75 7123.0 7181.50 7306.0
    aflppp_d82218 82800 20.0 7100.600000 231.401315 6758.0 6890.00 7116.0 7278.25 7525.0
    aflpp_d82218 82800 20.0 7092.400000 162.305560 6655.0 6987.00 7108.0 7240.25 7319.0
    aflplusplus 82800 20.0 5654.750000 254.510030 5328.0 5473.50 5532.0 5736.25 6197.0
    eclipser 82800 20.0 4890.400000 462.103247 3876.0 4703.25 4924.5 5103.75 5696.0
    fairfuzz 82800 20.0 780.450000 74.632768 497.0 747.75 807.5 818.25 826.0
    aflsmart 82800 20.0 784.000000 73.153767 497.0 805.00 806.5 818.00 819.0
    mopt 82800 20.0 770.300000 74.056948 497.0 740.75 806.5 815.00 820.0
    aflfast 82800 20.0 738.300000 115.000275 475.0 731.75 794.5 806.50 817.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

re2_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflppp_d82218 82800 20.0 2878.20 3.302312 2872.0 2875.75 2878.5 2880.00 2886.0
    aflplusplusplus 82800 20.0 2876.80 4.467426 2866.0 2875.00 2878.0 2880.00 2883.0
    aflplusplus 82800 20.0 2877.00 2.492093 2874.0 2875.00 2876.0 2878.25 2883.0
    aflpp_d82218 82800 20.0 2875.40 5.215362 2865.0 2872.00 2876.0 2880.00 2882.0
    hastefuzz 82800 20.0 2875.00 6.496963 2850.0 2874.00 2876.0 2878.25 2881.0
    aflsmart 82800 20.0 2868.55 4.936172 2860.0 2865.75 2868.5 2872.50 2876.0
    eclipser 82800 20.0 2860.85 23.079439 2785.0 2861.50 2867.5 2872.25 2878.0
    aflfast 82800 20.0 2865.45 6.270021 2849.0 2864.50 2866.5 2870.00 2873.0
    fairfuzz 82800 20.0 2834.65 73.055332 2631.0 2856.25 2864.0 2869.00 2875.0
    mopt 82800 20.0 2856.30 7.947856 2843.0 2849.75 2858.5 2862.25 2869.0
    aflrustrust 82800 20.0 2854.55 6.082546 2844.0 2851.75 2854.0 2857.25 2867.0
    libafl 82800 20.0 2848.90 9.883000 2820.0 2844.50 2849.5 2854.00 2864.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

sqlite3_ossfuzz summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflpp_d82218 82800 20.0 20343.05 141.148103 20114.0 20278.25 20344.0 20458.50 20565.0
    aflppp_d82218 82800 20.0 20234.40 140.122653 19849.0 20192.75 20241.0 20304.50 20484.0
    aflsmart 82800 20.0 19264.55 284.391590 18652.0 19128.25 19284.0 19399.00 19883.0
    eclipser 82800 20.0 19215.25 317.380707 18729.0 18953.50 19191.5 19428.00 19939.0
    mopt 82800 20.0 19038.10 288.714191 18474.0 18934.50 19060.5 19248.75 19483.0
    aflfast 82800 20.0 18711.00 281.495069 18104.0 18609.25 18713.5 18918.25 19183.0
    libafl 82800 20.0 18522.75 101.281720 18354.0 18454.50 18517.5 18569.25 18727.0
    aflplusplus 82800 20.0 17159.35 1270.044934 14839.0 16185.25 17651.5 18030.50 18771.0
    hastefuzz 82800 20.0 17116.55 1788.616739 13601.0 15740.00 17373.5 18584.00 19199.0
    aflplusplusplus 82800 20.0 16462.15 1372.310088 13961.0 15261.25 16548.0 17317.75 19217.0
    aflrustrust 82800 20.0 16031.60 1248.904463 13361.0 15406.25 16418.0 17010.50 17569.0
    fairfuzz 82800 20.0 13246.65 1692.282646 10991.0 11657.25 13522.5 14776.25 16178.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

stb_stbi_read_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libafl 82800 20.0 2178.45 44.681887 2111.0 2135.50 2192.0 2197.25 2255.0
    hastefuzz 82800 20.0 2153.55 51.157625 2086.0 2111.00 2129.5 2209.00 2218.0
    aflplusplusplus 82800 20.0 2145.45 56.193720 2022.0 2114.75 2117.5 2211.50 2223.0
    aflpp_d82218 82800 20.0 2150.95 63.886804 2026.0 2110.00 2116.0 2211.50 2298.0
    aflppp_d82218 82800 20.0 2128.30 34.919607 2111.0 2113.00 2115.0 2117.00 2222.0
    aflrustrust 82800 20.0 2137.00 66.105179 2021.0 2102.00 2113.0 2189.75 2290.0
    eclipser 82800 20.0 2109.45 30.165290 2080.0 2098.50 2106.0 2109.25 2192.0
    aflplusplus 82800 20.0 2067.55 66.216928 1986.0 2011.00 2054.5 2111.25 2212.0
    aflsmart 82800 20.0 2016.00 29.568831 1983.0 2004.00 2005.5 2008.00 2090.0
    mopt 82800 20.0 2003.60 38.368573 1953.0 1985.50 1994.0 2004.25 2088.0
    aflfast 82800 20.0 1983.90 30.450823 1903.0 1974.00 1984.0 2000.25 2065.0
    fairfuzz 82800 20.0 1967.25 49.496278 1854.0 1938.50 1983.0 1996.50 2086.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

vorbis_decode_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflpp_d82218 82800 20.0 1266.65 2.719810 1262.0 1264.00 1266.5 1268.00 1273.0
    aflrustrust 82800 20.0 1265.75 2.359193 1261.0 1264.50 1266.0 1267.00 1270.0
    hastefuzz 82800 20.0 1265.40 2.962929 1260.0 1264.00 1265.0 1267.00 1272.0
    aflplusplus 82800 20.0 1264.85 2.455391 1259.0 1263.75 1264.5 1266.00 1269.0
    aflppp_d82218 82800 20.0 1264.25 3.226127 1256.0 1262.75 1264.5 1266.25 1270.0
    aflplusplusplus 82800 20.0 1262.90 3.143916 1257.0 1260.75 1263.0 1264.00 1270.0
    aflsmart 82800 20.0 1255.00 2.919986 1250.0 1253.00 1254.5 1257.25 1262.0
    mopt 82800 20.0 1250.20 12.722132 1199.0 1250.00 1254.0 1255.25 1258.0
    aflfast 82800 20.0 1246.80 15.340349 1196.0 1248.50 1252.0 1254.00 1256.0
    eclipser 82800 20.0 1251.95 4.358597 1244.0 1249.75 1252.0 1255.25 1259.0
    libafl 82800 20.0 1253.50 4.968321 1246.0 1251.50 1252.0 1257.00 1267.0
    fairfuzz 82800 20.0 1234.70 28.252806 1160.0 1222.50 1249.5 1253.50 1258.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

experiment data

You can download the raw data for this report here.

Check out the documentation on how to create customized reports using this data. Also see some example Colab notebooks for doing custom analysis on the data here.

Experiment Description:

(None,)