FuzzBench: SBFT'23 Final Evaluation report

38 Coverage-based Benchmarks

Experiment Summary

We show two different aggregates (cross-benchmark) rankings of fuzzers. The first ranking is based on the average of per-benchmark scores, where the score represents the percentage of the highest reached median code coverage on a given benchmark (the higher, the better), and ties are broken by the average time taken to find the input. Repeatedly triggering the bug will not gain an extra score. The second ranking shows the average rank of fuzzers, after we rank them on each benchmark according to their median reached code coverage (the lower, the better).
By avg. score
average normalized score
fuzzer
hastefuzz 97.51
aflplusplusplus 94.19
aflplusplus 93.46
aflrustrust 91.82
libafl_libfuzzer 89.88
honggfuzz 88.08
pastis 86.70
libfuzzer 83.43
symsan 79.09
afl 75.60
aflsmart_plusplus 66.60
learnperffuzz 42.95
By avg. rank
average rank
fuzzer
hastefuzz 3.58
libafl_libfuzzer 3.66
aflplusplusplus 4.26
aflplusplus 4.82
aflrustrust 5.29
symsan 5.82
honggfuzz 6.03
pastis 6.55
libfuzzer 6.76
aflsmart_plusplus 7.71
afl 7.89
learnperffuzz 10.82
  • Critical difference diagram
    The diagram visualizes the average rank of fuzzers (second ranking above) while showing the significance of the differences as well. What is considered a "critical difference" (CD) is based on the Friedman/Nemenyi post-hoc test. See more in the documentation.
    Note: If a fuzzer does not support all benchmarks, its ranking as shown in this diagram can be lower than it should be. So please check the list of supported benchmarks for the fuzzer(s) of your interest. The list could be specified in the fuzzer's README.md like this.
  • Median relative code-coverages on each benchmark

    Note: The relative coverage summary table shows the median relative performance of each fuzzer to the experiment maximum. Thus the highest relative performance may not be 100%.
    trial_relative_coverage = trial_coverage / experiment_max_coverage

      hastefuzz aflplusplusplus aflplusplus aflrustrust libafl_libfuzzer honggfuzz pastis libfuzzer symsan afl aflsmart_plusplus learnperffuzz
    FuzzerMedian 98.00 98.00 97.50 97.50 98.00 97.00 92.50 89.00 96.00 95.00 90.50 44.50
    FuzzerMean 94.18 91.11 90.24 89.00 87.26 85.03 83.68 80.61 76.82 73.34 64.29 41.92
    arduinojson_json_fuzzer 99.00 99.00 99.00 99.00 99.00 99.00 99.00 nan 99.00 99.00 nan 84.00
    astc-encoder_fuzz_astc_physical_to_symbolic 99.00 99.00 nan 99.00 99.00 79.00 99.00 99.00 99.00 99.00 99.00 99.00
    bloaty_fuzz_target 76.00 76.00 98.00 93.00 97.00 96.00 90.00 85.00 67.00 nan 95.00 68.00
    botan_tls_server 68.00 67.00 66.00 44.00 44.00 63.00 58.00 57.00 nan 47.00 58.00 0.00
    brotli_decode_fuzzer 99.00 100.00 99.00 100.00 100.00 nan nan nan 99.00 99.00 nan nan
    curl_curl_fuzzer_http 98.00 98.00 97.00 97.00 97.00 99.00 94.00 88.00 97.00 95.00 92.00 68.00
    double-conversion_string_to_double_fuzzer 98.00 98.00 98.00 98.00 99.00 97.00 98.00 nan 99.00 96.00 nan nan
    draco_draco_pc_decoder_fuzzer 88.00 86.00 87.00 58.00 0.00 84.00 87.00 54.00 72.00 0.00 0.00 0.00
    dropbear_fuzzer-postauth_nomaths 98.00 98.00 96.00 96.00 1.00 99.00 nan 96.00 nan 78.00 78.00 1.00
    firestore_firestore_serializer_fuzzer 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 nan 100.00 100.00 1.00
    fmt_chrono-duration-fuzzer 99.00 99.00 98.00 99.00 99.00 99.00 99.00 99.00 99.00 98.00 97.00 12.00
    freetype2_ftfuzzer 93.00 95.00 81.00 82.00 72.00 90.00 77.00 70.00 92.00 65.00 64.00 22.00
    guetzli_guetzli_fuzzer 99.00 99.00 99.00 99.00 99.00 97.00 98.00 96.00 98.00 98.00 99.00 91.00
    harfbuzz_hb-shape-fuzzer 98.00 98.00 98.00 98.00 99.00 95.00 82.00 85.00 98.00 97.00 96.00 75.00
    icu_unicode_string_codepage_create_fuzzer 99.00 99.00 99.00 100.00 100.00 98.00 100.00 100.00 100.00 99.00 99.00 0.00
    jansson_json_load_dump_fuzzer 99.00 99.00 99.00 98.00 100.00 99.00 99.00 100.00 nan 99.00 99.00 26.00
    jsoncpp_jsoncpp_fuzzer 99.00 99.00 99.00 98.00 99.00 99.00 99.00 100.00 99.00 98.00 98.00 93.00
    lcms_cms_transform_fuzzer 85.00 76.00 68.00 88.00 93.00 65.00 82.00 75.00 95.00 36.00 77.00 19.00
    libaom_av1_dec_fuzzer 98.00 95.00 97.00 97.00 98.00 97.00 98.00 89.00 96.00 95.00 96.00 82.00
    libcoap_pdu_parse_fuzzer 98.00 98.00 98.00 98.00 91.00 99.00 99.00 99.00 nan 90.00 90.00 0.00
    libhevc_hevc_dec_fuzzer 99.00 99.00 99.00 99.00 99.00 99.00 99.00 98.00 nan nan nan nan
    libjpeg-turbo_libjpeg_turbo_fuzzer 99.00 99.00 99.00 99.00 99.00 99.00 99.00 99.00 99.00 99.00 99.00 98.00
    libpcap_fuzz_both 91.00 92.00 90.00 85.00 82.00 89.00 84.00 82.00 91.00 1.00 1.00 1.00
    libpng_libpng_read_fuzzer 94.00 95.00 94.00 94.00 95.00 95.00 89.00 95.00 95.00 94.00 94.00 66.00
    libxml2_xml 97.00 97.00 97.00 98.00 98.00 99.00 91.00 92.00 96.00 97.00 97.00 79.00
    libxslt_xpath 98.00 98.00 98.00 96.00 97.00 97.00 52.00 78.00 49.00 95.00 96.00 66.00
    mbedtls_fuzz_dtlsclient 70.00 70.00 69.00 70.00 87.00 68.00 66.00 68.00 70.00 68.00 69.00 47.00
    openh264_decoder_fuzzer 99.00 99.00 99.00 99.00 99.00 99.00 95.00 96.00 98.00 99.00 nan 87.00
    openssl_x509 99.00 99.00 99.00 99.00 99.00 98.00 99.00 98.00 99.00 99.00 99.00 98.00
    openthread_ot-ip6-send-fuzzer 82.00 77.00 74.00 86.00 88.00 76.00 73.00 77.00 76.00 71.00 72.00 0.00
    proj4_proj_crs_to_crs_fuzzer 87.00 86.00 68.00 60.00 9.00 97.00 89.00 89.00 80.00 9.00 9.00 3.00
    re2_fuzzer 99.00 99.00 99.00 98.00 99.00 98.00 98.00 99.00 99.00 nan nan 0.00
    sqlite3_ossfuzz 90.00 88.00 91.00 80.00 96.00 79.00 87.00 61.00 75.00 96.00 96.00 42.00
    stb_stbi_read_fuzzer 93.00 91.00 90.00 91.00 95.00 91.00 91.00 85.00 92.00 86.00 85.00 72.00
    systemd_fuzz-link-parser 99.00 99.00 94.00 92.00 94.00 96.00 53.00 72.00 99.00 90.00 91.00 58.00
    vorbis_decode_fuzzer 99.00 99.00 99.00 99.00 99.00 97.00 97.00 98.00 99.00 98.00 98.00 59.00
    woff2_convert_woff2ttf_fuzzer 97.00 97.00 97.00 96.00 98.00 nan 63.00 85.00 96.00 nan nan 76.00
    zlib_zlib_uncompress_fuzzer 97.00 nan 97.00 nan 97.00 nan 97.00 99.00 97.00 97.00 nan 0.00
    • Fuzzers are sorted by "FuzzerMean" (average median relative coverage), highest on the left.
    • Green background = highest relative median coverage.
    • Blue gradient background = greater than 95% relative median coverage.

arduinojson_json_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus 82800 20.0 401.60 0.502625 401.0 401.00 402.0 402.00 402.0
    aflplusplusplus 82800 20.0 401.60 0.502625 401.0 401.00 402.0 402.00 402.0
    pastis 82800 20.0 402.05 0.825578 400.0 402.00 402.0 403.00 403.0
    symsan 82800 20.0 402.00 0.000000 402.0 402.00 402.0 402.00 402.0
    honggfuzz 82800 20.0 401.45 1.099043 400.0 400.75 401.5 402.00 403.0
    aflrustrust 82800 20.0 401.35 0.489360 401.0 401.00 401.0 402.00 402.0
    hastefuzz 82800 20.0 401.45 0.510418 401.0 401.00 401.0 402.00 402.0
    libafl_libfuzzer 82800 20.0 400.70 0.571241 400.0 400.00 401.0 401.00 402.0
    afl 82800 20.0 399.55 1.316894 397.0 398.00 400.0 401.00 401.0
    learnperffuzz 82800 20.0 338.20 21.855025 253.0 336.50 342.0 350.25 357.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

astc-encoder_fuzz_astc_physical_to_symbolic summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 511.65 0.670820 511.0 511.0 512.0 512.00 513.0
    aflsmart_plusplus 82800 20.0 512.05 0.825578 511.0 511.0 512.0 513.00 513.0
    symsan 82800 20.0 512.35 0.670820 511.0 512.0 512.0 513.00 513.0
    aflplusplusplus 82800 20.0 511.65 0.745160 511.0 511.0 511.5 512.00 513.0
    aflrustrust 82800 20.0 511.55 0.825578 511.0 511.0 511.0 512.00 513.0
    hastefuzz 82800 20.0 511.45 0.759155 511.0 511.0 511.0 512.00 513.0
    learnperffuzz 82800 20.0 511.35 0.587143 511.0 511.0 511.0 512.00 513.0
    libafl_libfuzzer 82800 20.0 511.40 0.680557 511.0 511.0 511.0 512.00 513.0
    libfuzzer 82800 20.0 511.00 0.000000 511.0 511.0 511.0 511.00 511.0
    pastis 82800 20.0 511.00 0.000000 511.0 511.0 511.0 511.00 511.0
    honggfuzz 82800 20.0 408.50 8.444868 406.0 406.0 406.0 406.75 444.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

bloaty_fuzz_target summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus 82800 20.0 6300.95 122.778138 6055.0 6197.25 6353.5 6396.25 6436.0
    libafl_libfuzzer 82800 20.0 6299.10 78.765408 6165.0 6249.00 6285.0 6342.50 6444.0
    honggfuzz 82800 20.0 6172.15 140.333541 5903.0 6114.25 6221.0 6279.25 6308.0
    aflsmart_plusplus 82800 20.0 6132.15 121.333067 5958.0 6027.00 6126.5 6231.75 6332.0
    aflrustrust 82800 20.0 6054.55 73.172813 5958.0 5990.00 6038.0 6118.25 6177.0
    pastis 82800 20.0 5838.20 110.841565 5683.0 5751.25 5825.0 5895.25 6094.0
    libfuzzer 82800 20.0 5503.70 92.434386 5338.0 5441.25 5496.5 5579.75 5680.0
    hastefuzz 82800 20.0 4889.10 113.619355 4611.0 4820.75 4918.5 4980.25 5020.0
    aflplusplusplus 82800 20.0 4905.55 95.413491 4744.0 4872.50 4911.0 4951.50 5152.0
    learnperffuzz 82800 20.0 4438.00 97.926396 4363.0 4368.25 4389.5 4458.25 4687.0
    symsan 82800 20.0 4363.00 0.000000 4363.0 4363.00 4363.0 4363.00 4363.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

botan_tls_server summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    hastefuzz 82800 20.0 1901.95 425.474160 1167.0 1755.50 1945.0 2279.75 2377.0
    aflplusplusplus 82800 20.0 1755.35 376.569037 1175.0 1337.25 1890.5 1967.00 2351.0
    aflplusplus 82800 20.0 1796.95 471.568098 1244.0 1247.00 1886.5 2168.50 2791.0
    honggfuzz 82800 20.0 1830.50 136.220989 1549.0 1768.25 1796.0 1851.75 2214.0
    aflsmart_plusplus 82800 20.0 1628.85 336.988720 1241.0 1260.75 1660.5 1904.25 2166.0
    pastis 82800 20.0 1685.35 394.858839 1252.0 1269.00 1650.5 1975.75 2334.0
    libfuzzer 82800 20.0 1585.30 210.889869 1247.0 1415.25 1611.5 1773.25 1870.0
    afl 82800 20.0 1520.60 391.298567 1244.0 1245.75 1327.0 1756.50 2819.0
    libafl_libfuzzer 82800 20.0 1439.25 327.611913 1242.0 1243.75 1246.0 1634.25 2252.0
    aflrustrust 82800 20.0 1453.60 340.344779 1242.0 1244.00 1245.0 1792.75 2266.0
    learnperffuzz 82800 20.0 7.00 0.000000 7.0 7.00 7.0 7.00 7.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

brotli_decode_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplusplus 82800 20.0 903.35 0.933302 902.0 902.0 904.0 904.0 904.0
    aflrustrust 82800 20.0 903.30 0.978721 902.0 902.0 904.0 904.0 904.0
    libafl_libfuzzer 82800 20.0 903.60 0.820783 902.0 904.0 904.0 904.0 904.0
    afl 82800 20.0 902.85 1.871532 896.0 903.0 903.0 904.0 904.0
    aflplusplus 82800 20.0 902.90 1.209611 900.0 902.0 903.0 904.0 904.0
    hastefuzz 82800 20.0 902.65 1.225819 900.0 902.0 902.0 904.0 904.0
    symsan 82800 20.0 902.20 1.823819 896.0 902.0 902.0 904.0 904.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

curl_curl_fuzzer_http summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    honggfuzz 82800 20.0 10823.20 63.616946 10683.0 10797.25 10830.0 10875.00 10928.0
    aflplusplusplus 82800 20.0 10772.40 53.764986 10647.0 10745.75 10770.0 10795.50 10902.0
    hastefuzz 82800 20.0 10752.35 58.914099 10650.0 10716.50 10749.5 10785.25 10892.0
    libafl_libfuzzer 82800 20.0 10694.60 31.977624 10651.0 10673.25 10689.5 10698.50 10780.0
    symsan 82800 20.0 10684.25 42.499381 10615.0 10651.00 10674.5 10718.50 10765.0
    aflrustrust 82800 20.0 10640.80 63.048519 10516.0 10598.00 10646.0 10673.75 10746.0
    aflplusplus 82800 20.0 10675.15 87.724614 10524.0 10635.00 10645.5 10710.75 10914.0
    afl 82800 20.0 10415.40 59.813218 10345.0 10375.25 10396.5 10450.50 10562.0
    pastis 82800 20.0 10369.70 72.266393 10248.0 10328.50 10368.5 10392.25 10501.0
    aflsmart_plusplus 82800 20.0 10060.75 48.860570 9953.0 10044.00 10055.0 10103.50 10133.0
    libfuzzer 82800 20.0 9634.05 361.605845 9026.0 9425.50 9640.0 9920.50 10257.0
    learnperffuzz 82800 20.0 7409.25 221.635043 6926.0 7280.25 7443.0 7599.75 7716.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

double-conversion_string_to_double_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    symsan 82800 20.0 510.55 2.163696 507.0 509.00 510.5 511.25 514.0
    libafl_libfuzzer 82800 20.0 509.50 0.945905 508.0 508.75 510.0 510.00 511.0
    aflplusplus 82800 20.0 507.25 2.022895 503.0 506.00 507.5 509.00 510.0
    aflplusplusplus 82800 20.0 506.70 1.559352 504.0 505.75 507.0 507.25 510.0
    pastis 82800 20.0 506.40 1.759186 503.0 505.00 506.5 508.00 509.0
    hastefuzz 82800 20.0 506.45 1.431782 503.0 506.00 506.0 507.00 509.0
    aflrustrust 82800 20.0 504.55 1.669384 501.0 503.75 505.0 505.25 508.0
    honggfuzz 82800 20.0 502.50 1.538968 501.0 501.00 502.0 503.00 507.0
    afl 82800 20.0 498.10 2.245463 494.0 496.00 498.0 500.00 502.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

draco_draco_pc_decoder_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    hastefuzz 82800 20.0 1787.10 109.927201 1519.0 1741.25 1805.0 1827.25 1963.0
    pastis 82800 20.0 1751.70 240.146688 1344.0 1657.50 1799.5 1919.75 2046.0
    aflplusplus 82800 20.0 1810.70 84.101505 1675.0 1762.75 1796.5 1890.25 1973.0
    aflplusplusplus 82800 20.0 1807.50 84.069207 1712.0 1748.50 1769.0 1868.25 1974.0
    honggfuzz 82800 20.0 1734.95 96.087392 1486.0 1684.50 1731.5 1791.75 1926.0
    symsan 82800 20.0 1505.35 167.183566 1270.0 1357.50 1487.0 1634.00 1815.0
    aflrustrust 82800 20.0 1075.50 404.335778 142.0 934.50 1196.5 1358.25 1585.0
    libfuzzer 82800 20.0 1117.65 59.594485 948.0 1088.75 1110.0 1146.50 1215.0
    afl 82800 20.0 7.00 0.000000 7.0 7.00 7.0 7.00 7.0
    aflsmart_plusplus 82800 20.0 7.00 0.000000 7.0 7.00 7.0 7.00 7.0
    learnperffuzz 82800 20.0 7.00 0.000000 7.0 7.00 7.0 7.00 7.0
    libafl_libfuzzer 82800 20.0 7.00 0.000000 7.0 7.00 7.0 7.00 7.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

dropbear_fuzzer-postauth_nomaths summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    honggfuzz 82800 20.0 888.35 3.498496 882.0 886.75 889.0 890.00 896.0
    aflplusplusplus 82800 20.0 884.15 4.081731 875.0 881.00 884.0 887.25 891.0
    hastefuzz 82800 20.0 881.05 8.738752 859.0 880.00 883.5 887.00 892.0
    libfuzzer 82800 20.0 869.30 1.218282 868.0 868.00 869.0 871.00 871.0
    aflplusplus 82800 20.0 865.05 0.759155 863.0 865.00 865.0 865.00 867.0
    aflrustrust 82800 20.0 863.75 1.446411 861.0 862.00 864.0 865.00 866.0
    afl 82800 20.0 701.00 0.000000 701.0 701.00 701.0 701.00 701.0
    aflsmart_plusplus 82800 20.0 701.00 0.000000 701.0 701.00 701.0 701.00 701.0
    learnperffuzz 82800 20.0 44.55 154.512297 10.0 10.00 10.0 10.00 701.0
    libafl_libfuzzer 82800 20.0 10.00 0.000000 10.0 10.00 10.0 10.00 10.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

firestore_firestore_serializer_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 286.7 0.732695 285.0 287.0 287.0 287.0 287.0
    aflplusplus 82800 20.0 287.0 0.000000 287.0 287.0 287.0 287.0 287.0
    aflplusplusplus 82800 20.0 287.0 0.000000 287.0 287.0 287.0 287.0 287.0
    aflrustrust 82800 20.0 287.0 0.000000 287.0 287.0 287.0 287.0 287.0
    aflsmart_plusplus 82800 20.0 286.7 0.732695 285.0 287.0 287.0 287.0 287.0
    hastefuzz 82800 20.0 287.0 0.000000 287.0 287.0 287.0 287.0 287.0
    honggfuzz 82800 20.0 287.0 0.000000 287.0 287.0 287.0 287.0 287.0
    libafl_libfuzzer 82800 20.0 287.0 0.000000 287.0 287.0 287.0 287.0 287.0
    libfuzzer 82800 20.0 287.0 0.000000 287.0 287.0 287.0 287.0 287.0
    pastis 82800 20.0 287.0 0.000000 287.0 287.0 287.0 287.0 287.0
    learnperffuzz 82800 20.0 17.1 63.057117 3.0 3.0 3.0 3.0 285.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

fmt_chrono-duration-fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    symsan 82800 20.0 1038.55 244.835385 0.0 1092.50 1096.5 1099.25 1102.0
    hastefuzz 82800 20.0 1094.20 3.636077 1086.0 1091.00 1095.0 1097.00 1099.0
    aflplusplusplus 82800 20.0 1093.90 3.432047 1089.0 1090.00 1094.5 1096.00 1100.0
    libfuzzer 82800 20.0 1093.90 2.447340 1090.0 1092.75 1094.0 1096.00 1099.0
    aflrustrust 82800 20.0 1093.45 4.019361 1085.0 1091.00 1092.5 1096.25 1101.0
    libafl_libfuzzer 82800 20.0 1092.30 2.735729 1087.0 1090.75 1092.0 1094.00 1098.0
    honggfuzz 82800 20.0 1091.05 3.859029 1085.0 1088.00 1091.0 1094.00 1097.0
    pastis 82800 20.0 1091.15 2.497894 1085.0 1090.00 1091.0 1092.00 1097.0
    aflplusplus 82800 20.0 1088.45 2.684752 1084.0 1086.75 1088.0 1090.00 1093.0
    afl 82800 20.0 1080.30 3.785012 1074.0 1078.00 1081.0 1082.25 1087.0
    aflsmart_plusplus 82800 20.0 1078.35 5.214100 1063.0 1075.50 1079.5 1082.25 1084.0
    learnperffuzz 82800 20.0 107.25 81.822578 0.0 0.00 134.0 148.25 299.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

freetype2_ftfuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplusplus 82800 20.0 11915.25 189.920447 11410.0 11860.00 11936.5 12024.75 12234.0
    hastefuzz 82800 20.0 11545.05 371.203799 10825.0 11348.25 11659.5 11787.75 12169.0
    symsan 82800 20.0 11407.20 422.665302 10622.0 11220.75 11563.0 11724.25 11858.0
    honggfuzz 82800 20.0 11310.10 536.579312 10506.0 11052.50 11300.0 11751.75 12499.0
    aflrustrust 82800 20.0 10206.60 320.199346 9537.0 10057.75 10295.5 10422.50 10708.0
    aflplusplus 82800 20.0 10249.20 287.423326 9829.0 10023.50 10238.5 10437.50 10878.0
    pastis 82800 20.0 9615.95 413.064475 8620.0 9476.50 9639.5 9803.25 10389.0
    libafl_libfuzzer 82800 20.0 9213.55 511.368903 8376.0 8789.75 9061.0 9657.50 10182.0
    libfuzzer 82800 20.0 8641.60 863.213604 6830.0 8113.75 8831.5 9136.75 10622.0
    afl 82800 20.0 8072.10 239.520332 7711.0 7810.00 8195.0 8276.00 8312.0
    aflsmart_plusplus 82800 20.0 7953.50 231.907536 7567.0 7716.75 8069.5 8143.00 8231.0
    learnperffuzz 82800 20.0 3007.10 428.230865 2834.0 2834.00 2835.0 2835.00 4240.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

guetzli_guetzli_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    hastefuzz 82800 20.0 1493.70 12.092190 1460.0 1488.00 1500.0 1502.25 1506.0
    aflplusplus 82800 20.0 1495.80 5.809249 1481.0 1495.50 1497.0 1499.25 1503.0
    aflplusplusplus 82800 20.0 1491.20 13.387347 1462.0 1486.75 1494.0 1501.25 1507.0
    aflrustrust 82800 20.0 1492.85 3.856300 1484.0 1491.50 1493.5 1495.00 1500.0
    libafl_libfuzzer 82800 20.0 1492.50 3.332456 1480.0 1492.00 1492.5 1494.00 1497.0
    aflsmart_plusplus 82800 20.0 1491.60 1.698296 1487.0 1491.75 1492.0 1492.25 1493.0
    pastis 82800 20.0 1487.30 4.143479 1481.0 1482.75 1488.5 1490.25 1495.0
    symsan 82800 20.0 1486.30 8.820431 1466.0 1480.50 1488.0 1491.50 1502.0
    afl 82800 20.0 1485.40 5.688400 1477.0 1480.00 1485.0 1491.25 1493.0
    honggfuzz 82800 20.0 1470.55 5.365337 1462.0 1466.75 1470.5 1473.50 1480.0
    libfuzzer 82800 20.0 1459.30 9.325122 1447.0 1454.00 1456.0 1464.25 1483.0
    learnperffuzz 82800 20.0 1383.75 3.354102 1383.0 1383.00 1383.0 1383.00 1398.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

harfbuzz_hb-shape-fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libafl_libfuzzer 82800 20.0 11093.25 28.293341 11029.0 11077.00 11090.0 11115.25 11136.0
    aflrustrust 82800 20.0 10869.15 366.685908 9763.0 10974.25 11013.5 11035.25 11068.0
    hastefuzz 82800 20.0 10937.70 120.602479 10479.0 10941.00 10981.0 10991.00 11021.0
    aflplusplus 82800 20.0 10960.90 36.458917 10896.0 10938.00 10967.0 10982.25 11043.0
    aflplusplusplus 82800 20.0 10958.40 56.814935 10882.0 10910.25 10952.0 11002.25 11046.0
    symsan 82800 20.0 10927.90 53.522155 10791.0 10904.75 10930.0 10963.50 11012.0
    afl 82800 20.0 10809.85 56.868523 10692.0 10776.75 10825.5 10843.25 10914.0
    aflsmart_plusplus 82800 20.0 10745.10 48.156434 10663.0 10709.75 10739.0 10775.25 10844.0
    honggfuzz 82800 20.0 10623.15 119.832899 10321.0 10625.00 10664.5 10695.00 10727.0
    libfuzzer 82800 20.0 9551.60 164.098302 9174.0 9468.00 9562.5 9679.25 9759.0
    pastis 82800 20.0 9168.35 91.744081 9027.0 9103.50 9145.5 9219.25 9352.0
    learnperffuzz 82800 20.0 8393.20 414.529232 7617.0 8252.75 8433.0 8696.50 9024.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

icu_unicode_string_codepage_create_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflrustrust 82800 20.0 1340.55 0.759155 1339.0 1340.00 1341.0 1341.0 1341.0
    libafl_libfuzzer 82800 20.0 1336.90 11.968819 1302.0 1341.00 1341.0 1341.0 1341.0
    libfuzzer 82800 20.0 1341.00 0.000000 1341.0 1341.00 1341.0 1341.0 1341.0
    pastis 82800 20.0 1332.60 15.759375 1302.0 1336.75 1341.0 1341.0 1341.0
    symsan 82800 20.0 1340.45 0.686333 1339.0 1340.00 1341.0 1341.0 1341.0
    aflplusplus 82800 20.0 1340.25 0.786398 1339.0 1340.00 1340.0 1341.0 1341.0
    aflplusplusplus 82800 20.0 1339.70 0.732695 1339.0 1339.00 1340.0 1340.0 1341.0
    hastefuzz 82800 20.0 1338.30 8.572293 1302.0 1340.00 1340.0 1341.0 1341.0
    afl 82800 20.0 1339.10 0.852242 1338.0 1338.75 1339.0 1340.0 1341.0
    aflsmart_plusplus 82800 20.0 1333.60 14.372488 1299.0 1338.00 1339.0 1340.0 1341.0
    honggfuzz 82800 20.0 1319.10 17.639370 1302.0 1302.00 1316.5 1336.0 1341.0
    learnperffuzz 82800 20.0 170.10 415.450789 0.0 0.00 0.0 0.0 1144.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

jansson_json_load_dump_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libafl_libfuzzer 82800 20.0 773.60 0.598243 772.0 773.0 774.0 774.0 774.0
    libfuzzer 82800 20.0 774.00 0.000000 774.0 774.0 774.0 774.0 774.0
    pastis 82800 20.0 772.40 0.753937 771.0 772.0 772.0 773.0 774.0
    honggfuzz 82800 20.0 770.60 1.095445 769.0 770.0 770.5 771.0 773.0
    aflplusplusplus 82800 20.0 768.50 1.317893 766.0 767.0 769.0 770.0 770.0
    hastefuzz 82800 20.0 769.00 1.337712 767.0 768.0 769.0 770.0 771.0
    aflplusplus 82800 20.0 768.10 1.252366 766.0 767.0 768.0 769.0 770.0
    afl 82800 20.0 767.30 1.174286 765.0 767.0 767.0 768.0 769.0
    aflsmart_plusplus 82800 20.0 766.35 1.268028 763.0 766.0 767.0 767.0 768.0
    aflrustrust 82800 20.0 766.15 1.348488 764.0 765.0 766.0 767.0 769.0
    learnperffuzz 82800 20.0 314.95 155.591393 206.0 206.0 206.0 473.0 597.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

jsoncpp_jsoncpp_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libfuzzer 82800 20.0 525.00 0.000000 525.0 525.00 525.0 525.00 525.0
    honggfuzz 82800 20.0 522.15 0.670820 521.0 522.00 522.0 522.00 524.0
    pastis 82800 20.0 522.00 1.521772 518.0 521.75 522.0 522.00 525.0
    aflplusplus 82800 20.0 519.75 0.444262 519.0 519.75 520.0 520.00 520.0
    aflplusplusplus 82800 20.0 519.85 0.366348 519.0 520.00 520.0 520.00 520.0
    hastefuzz 82800 20.0 519.80 0.523148 518.0 520.00 520.0 520.00 520.0
    libafl_libfuzzer 82800 20.0 519.65 0.489360 519.0 519.00 520.0 520.00 520.0
    symsan 82800 20.0 520.00 0.000000 520.0 520.00 520.0 520.00 520.0
    aflrustrust 82800 20.0 518.90 0.447214 518.0 519.00 519.0 519.00 520.0
    afl 82800 20.0 518.15 1.182103 516.0 517.00 518.5 519.00 520.0
    aflsmart_plusplus 82800 20.0 517.70 1.031095 516.0 517.00 517.5 519.00 519.0
    learnperffuzz 82800 20.0 395.15 198.288805 9.0 475.25 492.0 498.25 507.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

lcms_cms_transform_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    symsan 82800 20.0 2238.30 30.528847 2189.0 2221.25 2235.5 2256.00 2329.0
    libafl_libfuzzer 82800 20.0 2084.35 211.879379 1539.0 2111.25 2170.5 2201.00 2242.0
    aflrustrust 82800 20.0 2049.40 89.966894 1824.0 2011.50 2064.5 2100.00 2194.0
    hastefuzz 82800 20.0 1944.20 187.532060 1556.0 1885.75 1994.5 2082.75 2160.0
    pastis 82800 20.0 1860.10 175.916816 1464.0 1808.00 1912.0 1977.25 2062.0
    aflsmart_plusplus 82800 20.0 1751.60 111.621730 1519.0 1708.50 1796.0 1833.50 1868.0
    aflplusplusplus 82800 20.0 1514.95 468.648839 712.0 1119.00 1788.5 1842.25 2074.0
    libfuzzer 82800 20.0 1769.15 86.148754 1638.0 1701.25 1751.0 1828.25 1974.0
    aflplusplus 82800 20.0 1605.65 120.081233 1404.0 1521.75 1607.0 1706.25 1779.0
    honggfuzz 82800 20.0 1382.80 524.289253 696.0 725.25 1530.5 1798.25 2036.0
    afl 82800 20.0 1069.30 484.155814 648.0 669.75 859.5 1643.25 1807.0
    learnperffuzz 82800 20.0 301.00 221.227913 38.0 38.00 459.5 463.25 523.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libaom_av1_dec_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    pastis 82800 20.0 11201.900000 65.725025 11050.0 11182.00 11221.0 11244.75 11281.0
    hastefuzz 82800 20.0 11137.650000 193.070556 10666.0 11050.00 11211.5 11240.50 11355.0
    libafl_libfuzzer 82800 20.0 11186.900000 105.208015 11064.0 11090.00 11139.5 11296.50 11348.0
    aflplusplus 82800 20.0 11081.500000 175.301094 10730.0 10932.50 11084.0 11243.75 11281.0
    aflrustrust 82800 20.0 10985.200000 194.851522 10366.0 10998.50 11018.5 11050.25 11209.0
    honggfuzz 82800 20.0 11048.900000 117.464844 10819.0 10985.50 11016.0 11109.75 11264.0
    aflsmart_plusplus 82800 20.0 11022.400000 106.097969 10846.0 10957.75 10993.5 11067.75 11213.0
    symsan 82800 20.0 10923.800000 245.561140 10416.0 10756.75 10986.5 11069.25 11288.0
    afl 82800 20.0 10809.450000 118.796808 10546.0 10745.50 10847.0 10878.50 10973.0
    aflplusplusplus 82800 20.0 10846.450000 143.291192 10613.0 10785.75 10809.0 10877.50 11259.0
    libfuzzer 82800 19.0 10176.473684 47.461526 10073.0 10153.50 10171.0 10215.00 10247.0
    learnperffuzz 82800 20.0 9408.750000 8.064705 9396.0 9402.25 9409.5 9415.50 9421.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libcoap_pdu_parse_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libfuzzer 82800 20.0 825.05 1.145931 824.0 824.00 825.0 825.25 827.0
    honggfuzz 82800 20.0 824.25 0.786398 823.0 824.00 824.0 825.00 825.0
    pastis 82800 20.0 822.30 2.792848 816.0 822.00 822.5 824.25 826.0
    aflplusplusplus 82800 20.0 817.60 1.313893 814.0 817.00 818.0 818.00 819.0
    aflplusplus 82800 20.0 817.05 1.356272 814.0 816.00 817.0 818.00 819.0
    aflrustrust 82800 20.0 805.95 25.730332 746.0 815.25 817.0 817.00 818.0
    hastefuzz 82800 20.0 817.40 1.273206 814.0 817.00 817.0 818.25 819.0
    libafl_libfuzzer 82800 20.0 757.75 0.444262 757.0 757.75 758.0 758.00 758.0
    afl 82800 20.0 748.20 1.151658 746.0 747.75 748.0 749.00 751.0
    aflsmart_plusplus 82800 20.0 747.95 1.190975 746.0 747.00 748.0 749.00 750.0
    learnperffuzz 82800 20.0 150.90 268.570269 0.0 0.00 0.0 142.00 656.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libhevc_hevc_dec_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libafl_libfuzzer 82800 20.0 10374.00 5.684698 10364.0 10371.00 10373.5 10377.50 10386.0
    aflrustrust 82800 20.0 10352.25 19.267849 10291.0 10347.75 10355.5 10361.25 10378.0
    hastefuzz 82800 20.0 10347.35 26.583087 10259.0 10341.00 10354.0 10362.50 10374.0
    aflplusplus 82800 20.0 10352.15 15.211751 10310.0 10342.75 10353.0 10362.00 10379.0
    aflplusplusplus 82800 20.0 10341.50 29.038088 10263.0 10339.00 10349.0 10359.25 10375.0
    pastis 82800 20.0 10340.25 6.414416 10329.0 10335.75 10339.5 10346.25 10352.0
    honggfuzz 82800 20.0 10320.70 13.357828 10293.0 10313.50 10321.5 10328.50 10344.0
    libfuzzer 82800 20.0 10272.50 19.283208 10241.0 10263.25 10269.0 10283.75 10320.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libjpeg-turbo_libjpeg_turbo_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    symsan 82800 20.0 2549.25 1.409554 2547.0 2549.00 2549.0 2549.00 2553.0
    aflplusplus 82800 20.0 2549.05 2.211810 2546.0 2547.00 2548.0 2551.00 2552.0
    aflplusplusplus 82800 20.0 2548.00 1.486784 2546.0 2547.00 2548.0 2548.00 2553.0
    hastefuzz 82800 20.0 2548.35 2.084403 2546.0 2547.00 2548.0 2548.50 2553.0
    aflsmart_plusplus 82800 20.0 2545.90 0.640723 2545.0 2545.75 2546.0 2546.00 2547.0
    libafl_libfuzzer 82800 20.0 2546.20 0.894427 2546.0 2546.00 2546.0 2546.00 2550.0
    pastis 82800 20.0 2546.40 1.729009 2545.0 2545.00 2546.0 2546.25 2550.0
    libfuzzer 82800 20.0 2546.60 2.036509 2544.0 2545.00 2545.0 2549.00 2549.0
    afl 82800 20.0 2545.20 1.794729 2543.0 2544.00 2544.5 2546.00 2550.0
    aflrustrust 82800 20.0 2543.90 1.293709 2543.0 2543.00 2544.0 2544.00 2549.0
    honggfuzz 82800 20.0 2543.75 0.910465 2542.0 2543.00 2544.0 2544.00 2545.0
    learnperffuzz 82800 20.0 2517.00 0.000000 2517.0 2517.00 2517.0 2517.00 2517.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libpcap_fuzz_both summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplusplus 82800 20.0 2906.60 143.853141 2677.0 2775.25 2895.0 3036.75 3121.0
    hastefuzz 82800 20.0 2862.65 137.344850 2600.0 2786.00 2863.0 2955.25 3088.0
    symsan 82800 20.0 2829.15 126.754331 2560.0 2739.25 2854.5 2913.50 3022.0
    aflplusplus 82800 20.0 2799.15 105.950969 2534.0 2763.00 2823.0 2854.25 2945.0
    honggfuzz 82800 20.0 2837.35 125.376601 2596.0 2778.75 2799.5 2915.50 3106.0
    aflrustrust 82800 20.0 2685.95 130.734152 2481.0 2591.50 2659.5 2774.25 2977.0
    pastis 82800 20.0 2604.75 149.723561 2161.0 2545.75 2622.5 2683.25 2827.0
    libfuzzer 82800 20.0 2556.20 94.026088 2223.0 2531.00 2569.5 2609.00 2654.0
    libafl_libfuzzer 82800 20.0 2489.40 263.155027 1972.0 2418.00 2561.0 2641.00 2892.0
    afl 82800 20.0 38.40 4.523622 33.0 33.00 42.0 42.00 42.0
    aflsmart_plusplus 82800 20.0 36.60 4.523622 33.0 33.00 33.0 42.00 42.0
    learnperffuzz 82800 20.0 33.10 12.073895 0.0 33.00 33.0 42.00 42.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libpng_libpng_read_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libafl_libfuzzer 82800 20.0 2024.10 21.036059 2005.0 2017.50 2019.0 2025.50 2110.0
    libfuzzer 82800 20.0 2013.65 13.144040 1963.0 2017.00 2018.0 2018.00 2019.0
    symsan 82800 20.0 2016.50 20.730095 1995.0 2010.50 2015.0 2016.25 2100.0
    honggfuzz 82800 20.0 2024.50 28.178940 2009.0 2011.75 2013.0 2016.25 2091.0
    aflplusplusplus 82800 20.0 2007.60 20.666271 1982.0 2002.75 2006.0 2007.25 2089.0
    hastefuzz 82800 20.0 2001.20 5.346913 1988.0 2000.50 2003.0 2005.00 2006.0
    aflplusplus 82800 20.0 2000.75 3.581752 1992.0 1998.75 2002.0 2003.00 2006.0
    afl 82800 20.0 1993.50 8.255779 1972.0 1991.00 1994.5 1998.25 2005.0
    aflsmart_plusplus 82800 20.0 1987.05 9.965599 1971.0 1976.50 1992.0 1994.00 1999.0
    aflrustrust 82800 20.0 1991.25 10.924983 1978.0 1981.75 1987.5 2002.25 2007.0
    pastis 82800 20.0 1902.55 36.537616 1877.0 1880.00 1881.5 1919.00 1984.0
    learnperffuzz 82800 20.0 1400.00 0.000000 1400.0 1400.00 1400.0 1400.00 1400.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libxml2_xml summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    honggfuzz 82800 20.0 15595.50 55.071724 15500.0 15555.75 15594.5 15635.00 15698.0
    libafl_libfuzzer 82800 20.0 15381.85 663.838937 12569.0 15503.25 15539.5 15554.00 15597.0
    aflrustrust 82800 20.0 15446.15 324.340347 14083.0 15475.50 15514.0 15555.00 15592.0
    aflsmart_plusplus 82800 20.0 15350.55 62.546383 15258.0 15289.25 15366.5 15389.75 15481.0
    afl 82800 20.0 15340.70 72.707127 15227.0 15281.50 15353.5 15390.25 15468.0
    aflplusplus 82800 20.0 15290.35 167.917672 14633.0 15267.75 15313.5 15374.75 15451.0
    hastefuzz 82800 20.0 15261.45 101.376046 15030.0 15223.50 15269.0 15350.00 15402.0
    aflplusplusplus 82800 20.0 15274.90 126.543481 14993.0 15198.75 15240.0 15373.00 15545.0
    symsan 82800 20.0 14742.05 1480.830154 8556.0 15054.50 15158.5 15211.50 15365.0
    libfuzzer 82800 20.0 14539.75 148.650022 14270.0 14437.50 14568.5 14669.50 14743.0
    pastis 82800 20.0 14434.60 196.054343 14118.0 14274.25 14372.0 14615.75 14718.0
    learnperffuzz 82800 20.0 12114.65 1018.394158 8556.0 11903.00 12530.5 12661.00 12914.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libxslt_xpath summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus 82800 20.0 11133.10 61.574858 11023.0 11075.25 11159.0 11180.75 11233.0
    hastefuzz 82800 20.0 11098.90 55.420783 10982.0 11074.00 11107.5 11129.00 11197.0
    aflplusplusplus 82800 20.0 11082.85 68.674882 10917.0 11039.25 11079.0 11123.75 11204.0
    libafl_libfuzzer 82800 20.0 11091.25 89.251198 10984.0 11032.75 11073.0 11125.50 11300.0
    honggfuzz 82800 20.0 11014.35 100.897173 10827.0 10961.75 11027.5 11048.50 11204.0
    aflsmart_plusplus 82800 20.0 10930.55 44.096157 10849.0 10895.25 10925.0 10959.75 11027.0
    aflrustrust 82800 20.0 10867.55 57.394732 10726.0 10840.50 10855.5 10889.50 10986.0
    afl 82800 20.0 10839.70 46.305735 10752.0 10812.00 10834.0 10881.25 10905.0
    libfuzzer 82800 20.0 8923.00 313.564230 8583.0 8759.50 8862.5 8993.00 10120.0
    learnperffuzz 82800 20.0 7549.20 147.462609 7262.0 7487.75 7546.5 7619.75 7876.0
    pastis 82800 20.0 5956.45 8.929401 5950.0 5952.00 5954.0 5955.50 5989.0
    symsan 82800 20.0 7268.55 2538.256726 5649.0 5649.00 5649.0 11016.00 11123.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

mbedtls_fuzz_dtlsclient summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libafl_libfuzzer 82800 20.0 3296.30 464.647019 2739.0 2771.75 3466.0 3706.00 3946.0
    aflrustrust 82800 20.0 3087.70 419.942615 2719.0 2745.00 2796.5 3404.00 3821.0
    aflplusplusplus 82800 20.0 2772.00 32.887368 2720.0 2750.00 2771.0 2790.50 2835.0
    symsan 82800 20.0 2807.05 194.663400 2730.0 2745.50 2767.5 2773.75 3630.0
    hastefuzz 82800 20.0 2781.40 42.235430 2729.0 2757.75 2766.5 2803.25 2889.0
    aflplusplus 82800 20.0 2726.90 28.931680 2684.0 2709.50 2724.5 2733.00 2809.0
    aflsmart_plusplus 82800 20.0 2819.95 324.756180 2645.0 2711.00 2723.5 2729.00 3855.0
    afl 82800 20.0 2727.30 25.854044 2698.0 2716.25 2718.5 2726.00 2814.0
    libfuzzer 82800 20.0 2706.00 25.418601 2665.0 2689.00 2701.5 2716.00 2775.0
    honggfuzz 82800 20.0 2687.25 32.273298 2627.0 2664.25 2687.0 2703.50 2770.0
    pastis 82800 20.0 2629.75 25.606075 2576.0 2618.00 2634.5 2644.25 2685.0
    learnperffuzz 82800 20.0 1858.00 0.000000 1858.0 1858.00 1858.0 1858.00 1858.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

openh264_decoder_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libafl_libfuzzer 82800 20.0 9554.90 14.293539 9501.0 9553.00 9557.0 9559.25 9574.0
    aflrustrust 82800 20.0 9542.50 8.786952 9521.0 9537.50 9542.0 9550.00 9556.0
    honggfuzz 82800 20.0 9540.75 7.979414 9523.0 9536.50 9541.0 9548.00 9550.0
    afl 82800 20.0 9530.30 28.204330 9422.0 9525.25 9533.0 9540.25 9560.0
    hastefuzz 82800 20.0 9466.15 86.566236 9272.0 9424.25 9513.5 9524.25 9551.0
    aflplusplus 82800 20.0 9448.25 119.543815 9167.0 9418.25 9513.0 9529.00 9541.0
    aflplusplusplus 82800 20.0 9434.40 113.877774 9214.0 9338.50 9498.0 9517.50 9530.0
    symsan 82800 20.0 9368.30 101.474965 9156.0 9334.50 9384.0 9427.75 9505.0
    libfuzzer 82800 20.0 9207.70 24.406858 9173.0 9190.75 9200.0 9224.25 9265.0
    pastis 82800 20.0 9147.90 75.225206 8987.0 9115.25 9173.0 9194.75 9264.0
    learnperffuzz 82800 20.0 8365.50 312.934077 7986.0 7986.00 8368.0 8583.75 8931.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

openssl_x509 summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflrustrust 82800 20.0 5830.90 2.954034 5821.0 5830.0 5832.0 5833.00 5834.0
    libafl_libfuzzer 82800 20.0 5834.65 10.001447 5832.0 5832.0 5832.0 5832.50 5877.0
    aflsmart_plusplus 82800 20.0 5824.50 11.971457 5796.0 5826.0 5831.0 5831.00 5834.0
    aflplusplus 82800 20.0 5829.20 3.941680 5821.0 5826.0 5830.5 5832.00 5834.0
    hastefuzz 82800 20.0 5827.10 8.129867 5811.0 5821.0 5830.5 5834.00 5836.0
    afl 82800 20.0 5829.45 1.356272 5825.0 5829.0 5829.0 5830.25 5831.0
    aflplusplusplus 82800 20.0 5823.35 6.368301 5809.0 5819.5 5824.0 5826.75 5834.0
    symsan 82800 20.0 5826.00 4.701623 5819.0 5823.0 5824.0 5828.00 5835.0
    pastis 82800 20.0 5818.70 7.807554 5805.0 5812.5 5821.0 5825.00 5829.0
    libfuzzer 82800 20.0 5820.95 6.117060 5814.0 5816.0 5816.5 5827.25 5829.0
    honggfuzz 82800 20.0 5813.65 5.527682 5809.0 5809.0 5810.5 5819.50 5823.0
    learnperffuzz 82800 20.0 5774.00 0.000000 5774.0 5774.0 5774.0 5774.00 5774.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

openthread_ot-ip6-send-fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libafl_libfuzzer 82800 20.0 3362.00 262.845962 2831.0 3063.00 3515.0 3574.00 3619.0
    aflrustrust 82800 20.0 3343.70 246.072540 2979.0 3049.25 3466.0 3554.25 3575.0
    hastefuzz 82800 20.0 3314.15 257.679580 3044.0 3063.50 3297.5 3562.50 3597.0
    libfuzzer 82800 20.0 3123.35 115.911616 3071.0 3091.00 3102.0 3107.25 3612.0
    aflplusplusplus 82800 20.0 3240.50 240.581204 3040.0 3066.75 3078.5 3530.00 3595.0
    symsan 82800 20.0 3145.75 179.065079 3041.0 3050.00 3057.0 3095.00 3540.0
    honggfuzz 82800 20.0 3146.05 319.028043 2899.0 3021.75 3041.5 3059.25 3991.0
    aflplusplus 82800 20.0 2981.20 137.467546 2826.0 2884.00 2980.5 3061.25 3386.0
    pastis 82800 20.0 2954.45 61.228197 2907.0 2915.25 2921.5 2975.00 3097.0
    aflsmart_plusplus 82800 20.0 2880.00 45.029815 2817.0 2829.25 2909.0 2915.50 2931.0
    afl 82800 20.0 2891.85 128.556060 2820.0 2823.75 2867.0 2907.50 3408.0
    learnperffuzz 82800 20.0 0.00 0.000000 0.0 0.00 0.0 0.00 0.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

proj4_proj_crs_to_crs_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    honggfuzz 82800 20.0 7942.25 189.214408 7439.0 7884.75 7979.5 8078.50 8222.0
    pastis 82800 20.0 7360.85 199.906103 6938.0 7195.75 7387.5 7531.00 7666.0
    libfuzzer 82800 20.0 7381.95 93.830460 7155.0 7356.75 7385.0 7427.50 7570.0
    hastefuzz 82800 20.0 7160.10 155.682504 6908.0 7025.75 7200.0 7227.00 7486.0
    aflplusplusplus 82800 20.0 7078.55 192.403216 6653.0 7015.25 7097.0 7168.75 7516.0
    symsan 82800 20.0 6644.85 191.523882 6298.0 6527.00 6632.0 6794.25 7005.0
    aflplusplus 82800 20.0 5681.45 275.105430 5284.0 5483.75 5592.5 5787.25 6172.0
    aflrustrust 82800 20.0 4951.60 218.501210 4533.0 4883.25 4944.0 5034.50 5582.0
    aflsmart_plusplus 82800 20.0 809.15 20.438032 732.0 807.00 818.0 819.25 823.0
    afl 82800 20.0 771.70 99.925815 488.0 806.00 808.0 816.25 819.0
    libafl_libfuzzer 82800 20.0 791.20 74.546065 485.0 808.00 808.0 817.00 820.0
    learnperffuzz 82800 20.0 203.85 112.163544 55.0 55.00 280.5 283.00 298.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

re2_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libfuzzer 82800 20.0 2870.85 14.158037 2850.0 2853.00 2879.0 2882.00 2884.0
    aflplusplusplus 82800 20.0 2877.35 2.978431 2869.0 2876.00 2877.5 2880.00 2881.0
    hastefuzz 82800 20.0 2876.00 4.039281 2866.0 2874.00 2877.0 2878.00 2884.0
    libafl_libfuzzer 82800 20.0 2876.25 5.138861 2860.0 2874.50 2877.0 2879.25 2883.0
    symsan 82800 20.0 2877.05 5.276313 2868.0 2873.75 2877.0 2878.75 2887.0
    aflplusplus 82800 20.0 2875.85 3.717314 2867.0 2874.75 2875.5 2878.00 2881.0
    aflrustrust 82800 20.0 2855.10 6.239939 2844.0 2850.50 2854.5 2860.25 2867.0
    honggfuzz 82800 20.0 2852.95 7.514722 2837.0 2846.75 2854.5 2859.00 2863.0
    pastis 82800 20.0 2853.25 5.590170 2842.0 2849.00 2853.0 2858.25 2862.0
    learnperffuzz 82800 20.0 229.85 707.717924 0.0 0.00 0.0 0.00 2357.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

sqlite3_ossfuzz summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 19020.25 480.371991 17537.0 18853.00 19091.0 19346.25 19603.0
    aflsmart_plusplus 82800 20.0 19046.15 350.949880 18094.0 18906.75 19037.5 19254.00 19739.0
    libafl_libfuzzer 82800 20.0 18860.70 509.265012 17445.0 18831.00 18985.0 19137.00 19596.0
    aflplusplus 82800 20.0 17743.70 1361.709340 14835.0 17361.00 18015.0 18647.75 19440.0
    hastefuzz 82800 20.0 17457.60 1595.239048 13407.0 16632.75 17901.5 18830.25 19187.0
    aflplusplusplus 82800 20.0 17008.30 1456.385038 13998.0 15837.50 17379.0 18141.00 18821.0
    pastis 82800 20.0 17194.10 364.127204 16534.0 16988.00 17306.0 17473.75 17666.0
    aflrustrust 82800 20.0 15956.30 1032.571094 13962.0 15503.00 15874.5 16576.50 18133.0
    honggfuzz 82800 20.0 15637.05 431.400714 14780.0 15467.75 15710.5 15943.00 16346.0
    symsan 82800 20.0 14834.05 580.894906 13895.0 14358.25 14866.0 15337.75 15917.0
    libfuzzer 82800 20.0 12235.90 540.065873 11349.0 11881.00 12234.0 12500.25 13575.0
    learnperffuzz 82800 20.0 8683.20 750.354204 7949.0 8021.00 8444.5 9186.75 10598.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

stb_stbi_read_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libafl_libfuzzer 82800 20.0 2196.75 19.243249 2117.0 2199.75 2201.0 2203.25 2207.0
    hastefuzz 82800 20.0 2165.65 54.524041 2111.0 2112.00 2159.0 2214.50 2266.0
    symsan 82800 20.0 2156.45 70.760883 2029.0 2119.75 2124.0 2222.00 2307.0
    pastis 82800 20.0 2116.85 3.645834 2106.0 2115.00 2118.5 2119.00 2120.0
    honggfuzz 82800 20.0 2117.75 20.388012 2095.0 2111.75 2115.5 2116.00 2201.0
    aflplusplusplus 82800 20.0 2122.50 59.070431 2019.0 2111.75 2115.0 2141.50 2212.0
    aflrustrust 82800 20.0 2118.95 39.569359 2027.0 2106.00 2112.5 2130.50 2203.0
    aflplusplus 82800 20.0 2073.90 57.076034 2010.0 2016.25 2096.5 2113.00 2212.0
    afl 82800 20.0 2012.00 42.963877 1978.0 2001.75 2004.5 2007.25 2191.0
    aflsmart_plusplus 82800 20.0 1986.25 29.829119 1953.0 1962.50 1983.5 2000.50 2087.0
    libfuzzer 82800 20.0 1962.10 11.387435 1936.0 1956.00 1964.5 1971.50 1975.0
    learnperffuzz 82800 20.0 1682.55 6.931811 1681.0 1681.00 1681.0 1681.00 1712.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

vorbis_decode_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libafl_libfuzzer 82800 20.0 1270.95 1.848897 1267.0 1270.00 1271.0 1272.00 1274.0
    aflplusplus 82800 20.0 1265.85 2.641272 1262.0 1264.00 1265.5 1267.00 1271.0
    aflrustrust 82800 20.0 1260.40 15.550935 1199.0 1262.75 1265.0 1266.25 1268.0
    hastefuzz 82800 20.0 1265.10 1.916686 1262.0 1264.00 1265.0 1266.25 1269.0
    symsan 82800 20.0 1264.10 2.712544 1260.0 1262.50 1264.0 1265.25 1270.0
    aflplusplusplus 82800 20.0 1262.50 2.139848 1257.0 1261.75 1263.0 1264.00 1265.0
    libfuzzer 82800 20.0 1258.40 4.417668 1245.0 1256.75 1259.5 1262.00 1263.0
    aflsmart_plusplus 82800 20.0 1255.55 4.223680 1240.0 1255.00 1256.5 1258.00 1259.0
    afl 82800 20.0 1254.05 3.332061 1248.0 1251.75 1255.0 1255.25 1261.0
    honggfuzz 82800 20.0 1245.65 12.695938 1198.0 1243.50 1247.5 1252.25 1259.0
    pastis 82800 20.0 1246.75 6.051316 1235.0 1243.00 1244.0 1250.00 1258.0
    learnperffuzz 82800 20.0 774.40 68.870894 759.0 759.00 759.0 759.00 1067.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

woff2_convert_woff2ttf_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libafl_libfuzzer 82800 20.0 1187.25 11.172688 1164.0 1180.50 1188.5 1194.25 1206.0
    hastefuzz 82800 20.0 1177.90 11.827266 1142.0 1172.00 1179.0 1185.25 1193.0
    aflplusplus 82800 20.0 1174.90 11.782681 1153.0 1169.50 1172.0 1184.00 1195.0
    aflplusplusplus 82800 20.0 1173.05 12.054591 1153.0 1163.75 1171.5 1183.00 1197.0
    aflrustrust 82800 20.0 1168.05 13.593632 1142.0 1157.00 1168.5 1179.75 1190.0
    symsan 82800 20.0 1170.75 14.048319 1148.0 1159.75 1167.5 1182.00 1193.0
    libfuzzer 82800 20.0 1038.00 38.450241 979.0 1019.50 1030.5 1052.25 1151.0
    learnperffuzz 82800 20.0 899.40 63.180110 763.0 885.75 919.5 942.25 960.0
    pastis 82800 20.0 770.00 0.000000 770.0 770.00 770.0 770.00 770.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

zlib_zlib_uncompress_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libfuzzer 82800 20.0 466.550000 4.394674 459.0 463.00 467.5 471.00 472.0
    libafl_libfuzzer 82800 20.0 462.550000 5.933624 449.0 457.75 462.0 469.00 470.0
    hastefuzz 82800 20.0 460.600000 1.818038 456.0 460.00 460.5 462.00 463.0
    afl 82800 20.0 459.850000 4.804329 455.0 455.75 459.0 462.00 470.0
    aflplusplus 82800 20.0 460.150000 4.451789 456.0 457.00 459.0 462.00 470.0
    pastis 82800 20.0 459.350000 3.759549 450.0 459.00 459.0 459.50 470.0
    symsan 82800 19.0 459.736842 5.183758 451.0 456.00 459.0 462.00 469.0
    learnperffuzz 82800 20.0 109.600000 146.745896 0.0 0.00 0.0 226.75 354.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

experiment data

You can download the raw data for this report here.

Check out the documentation on how to create customized reports using this data. Also see some example Colab notebooks for doing custom analysis on the data here.

The experiment was conducted using this FuzzBench commit: git_hash

To reproduce this experiment run the following commands in your FuzzBench repo:
# Check out the right commit.
git checkout git_hash
# Download the internal config file.
curl https://storage.googleapis.com/fuzzbench-data/FuzzBench: SBFT'23 Final Evaluation/input/config/experiment.yaml > /tmp/experiment-config.yaml
make install-dependencies
# Launch the experiment using paramters from the internal config file.
PYTHONPATH=. python experiment/reproduce_experiment.py -c /tmp/experiment-config.yaml -e <new_experiment_name>

Experiment Description:

from cached data