FuzzBench: 2023-06-13-gmfuzzer report

(experiment incomplete/still running...)

experiment summary

We show two different aggregate (cross-benchmark) rankings of fuzzers. The first is based on the average of per-benchmarks scores, where the score represents the percentage of the highest reached median code-coverage on a given benchmark (higher value is better). The second ranking shows the average rank of fuzzers, after we rank them on each benchmark according to their median reached code-covereges (lower value is better).
By avg. score
average normalized score
fuzzer
gmfuzzer_honggfuzz 95.97
gmfuzzer_236 90.84
gmfuzzer 90.71
gmfuzzer_133 90.50
ecofuzz 79.85
gmfuzzer_aflplusplus 79.31
gmfuzzer_libfuzzer 71.55
gmfuzzer_mopt 66.67
gmfuzzer_ecofuzz 63.26
gmfuzzer_fairfuzz 60.92
gmfuzzer_darwin 39.97
darwin 37.35
pastis 32.20
By avg. rank
average rank
fuzzer
gmfuzzer_236 2.35
gmfuzzer 2.96
gmfuzzer_133 3.13
gmfuzzer_libfuzzer 5.39
gmfuzzer_honggfuzz 5.43
gmfuzzer_aflplusplus 5.48
gmfuzzer_fairfuzz 7.65
gmfuzzer_mopt 7.74
gmfuzzer_darwin 8.61
gmfuzzer_ecofuzz 9.00
ecofuzz 9.09
pastis 9.17
darwin 9.26
  • Critical difference diagram
    The diagram visualizes the average rank of fuzzers (second ranking above) while showing the significance of the differences as well. What is considered a "critical difference" (CD) is based on the Friedman/Nemenyi post-hoc test. See more in the documentation.
    Note: If a fuzzer does not support all benchmarks, its ranking as shown in this diagram can be lower than it should be. So please check the list of supported benchmarks for the fuzzer(s) of your interest. The list could be specified in the fuzzer's README.md like this.
  • Median relative code-coverages on each benchmark

    Note: The relative coverage summary table shows the median relative performance of each fuzzer to the experiment maximum. Thus the highest relative performance may not be 100%.
    trial_relative_coverage = trial_coverage / experiment_max_coverage

      gmfuzzer_236 gmfuzzer gmfuzzer_133 darwin gmfuzzer_libfuzzer gmfuzzer_honggfuzz gmfuzzer_aflplusplus gmfuzzer_darwin pastis gmfuzzer_mopt ecofuzz gmfuzzer_fairfuzz gmfuzzer_ecofuzz
    FuzzerMedian 97.00 96.00 96.00 97.00 96.00 94.00 93.00 95.50 84.00 93.00 88.00 88.50 86.00
    FuzzerMean 95.10 95.05 94.71 92.89 92.35 92.00 91.16 89.30 77.89 77.47 76.78 74.56 73.37
    bloaty_fuzz_target 95.00 95.00 94.00 nan 91.00 92.00 91.00 nan 89.00 nan 89.00 77.00 82.00
    curl_curl_fuzzer_http 98.00 98.00 98.00 nan 96.00 94.00 99.00 nan nan 94.00 93.00 94.00 93.00
    freetype2_ftfuzzer 92.00 94.00 94.00 nan nan 90.00 82.00 nan nan 59.00 59.00 56.00 53.00
    harfbuzz_hb-shape-fuzzer 96.00 96.00 96.00 nan nan 91.00 96.00 78.00 nan 93.00 80.00 85.00 83.00
    jsoncpp_jsoncpp_fuzzer 100.00 100.00 100.00 98.00 100.00 100.00 nan 99.00 nan 98.00 98.00 98.00 98.00
    lcms_cms_transform_fuzzer 89.00 92.00 89.00 70.00 88.00 86.00 84.00 71.00 84.00 39.00 38.00 46.00 40.00
    libjpeg-turbo_libjpeg_turbo_fuzzer 99.00 100.00 99.00 nan 100.00 99.00 99.00 nan nan nan 96.00 96.00 89.00
    libpcap_fuzz_both 95.00 93.00 93.00 nan 82.00 92.00 89.00 nan 94.00 1.00 1.00 1.00 1.00
    libpng_libpng_read_fuzzer 96.00 96.00 96.00 94.00 96.00 96.00 96.00 95.00 nan 95.00 92.00 nan 93.00
    libxml2_xml nan nan nan 97.00 nan 97.00 98.00 98.00 nan 97.00 75.00 90.00 86.00
    libxslt_xpath 99.00 99.00 99.00 94.00 nan 97.00 nan 96.00 52.00 94.00 86.00 96.00 88.00
    mbedtls_fuzz_dtlsclient 86.00 85.00 86.00 nan 85.00 81.00 87.00 nan nan 85.00 84.00 87.00 84.00
    openh264_decoder_fuzzer 99.00 99.00 99.00 nan nan 99.00 99.00 nan nan 97.00 90.00 93.00 95.00
    openssl_x509 99.00 99.00 99.00 99.00 99.00 99.00 nan 99.00 nan 99.00 99.00 99.00 99.00
    openthread_ot-ip6-send-fuzzer 80.00 79.00 78.00 nan 71.00 70.00 80.00 nan 70.00 66.00 66.00 66.00 65.00
    proj4_proj_crs_to_crs_fuzzer 90.00 89.00 89.00 nan 83.00 87.00 83.00 nan nan 9.00 9.00 10.00 9.00
    re2_fuzzer 99.00 99.00 99.00 99.00 99.00 99.00 nan 99.00 99.00 99.00 98.00 nan nan
    sqlite3_ossfuzz nan nan nan 97.00 nan 71.00 75.00 72.00 nan 72.00 64.00 62.00 55.00
    stb_stbi_read_fuzzer 93.00 95.00 93.00 88.00 91.00 93.00 92.00 86.00 nan nan 85.00 nan nan
    systemd_fuzz-link-parser 99.00 99.00 99.00 nan 99.00 97.00 97.00 nan 53.00 90.00 88.00 91.00 86.00
    vorbis_decode_fuzzer 97.00 93.00 93.00 nan 93.00 93.00 93.00 nan nan nan 91.00 nan nan
    woff2_convert_woff2ttf_fuzzer 98.00 98.00 98.00 nan 98.00 96.00 97.00 nan 62.00 91.00 89.00 nan nan
    zlib_zlib_uncompress_fuzzer 98.00 98.00 98.00 nan 99.00 97.00 95.00 nan 98.00 94.00 96.00 95.00 95.00
    • Fuzzers are sorted by "FuzzerMean" (average median relative coverage), highest on the left.
    • Green background = highest relative median coverage.
    • Blue gradient background = greater than 95% relative median coverage.

bloaty_fuzz_target summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    gmfuzzer_236 51300 12.0 6210.916667 106.222287 6003.0 6151.75 6225.0 6284.75 6346.0
    gmfuzzer 51300 13.0 6218.615385 102.508161 6069.0 6134.00 6202.0 6272.00 6413.0
    gmfuzzer_133 51300 14.0 6155.857143 104.324831 5976.0 6113.50 6165.0 6184.75 6328.0
    gmfuzzer_honggfuzz 51300 15.0 6027.333333 99.268515 5873.0 5949.50 6038.0 6105.50 6172.0
    gmfuzzer_aflplusplus 51300 19.0 5943.578947 106.172457 5754.0 5882.00 5950.0 6030.00 6098.0
    gmfuzzer_libfuzzer 51300 20.0 5932.100000 116.087490 5743.0 5829.50 5940.5 5990.00 6174.0
    pastis 51300 9.0 5775.222222 123.990703 5596.0 5687.00 5807.0 5821.00 5937.0
    ecofuzz 51300 20.0 5743.550000 192.538028 5223.0 5736.75 5806.0 5863.00 5933.0
    gmfuzzer_ecofuzz 51300 4.0 5355.500000 24.583192 5331.0 5338.50 5352.5 5369.50 5386.0
    gmfuzzer_fairfuzz 51300 5.0 5050.200000 99.132235 4908.0 5033.00 5056.0 5068.00 5186.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

curl_curl_fuzzer_http summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    gmfuzzer_aflplusplus 82800 1.0 10944.000000 NaN 10944.0 10944.00 10944.0 10944.00 10944.0
    gmfuzzer_133 82800 20.0 10854.150000 67.450001 10758.0 10807.75 10844.0 10879.75 11005.0
    gmfuzzer_236 82800 18.0 10824.277778 66.807718 10674.0 10795.75 10826.0 10852.25 10948.0
    gmfuzzer 82800 20.0 10835.700000 55.184952 10772.0 10793.50 10817.5 10866.75 10947.0
    gmfuzzer_libfuzzer 82800 19.0 10657.000000 51.052261 10573.0 10618.50 10640.0 10697.50 10746.0
    gmfuzzer_fairfuzz 82800 19.0 10395.105263 61.063623 10227.0 10376.50 10418.0 10431.00 10474.0
    gmfuzzer_mopt 82800 19.0 10386.842105 105.563179 10001.0 10363.50 10407.0 10428.50 10506.0
    gmfuzzer_honggfuzz 82800 19.0 10351.894737 122.585977 10072.0 10305.00 10376.0 10444.00 10495.0
    gmfuzzer_ecofuzz 82800 17.0 10317.764706 57.498402 10128.0 10309.00 10329.0 10350.00 10376.0
    ecofuzz 82800 19.0 10247.842105 100.149479 9896.0 10224.00 10254.0 10293.00 10385.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

freetype2_ftfuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    gmfuzzer_133 54000 6.0 12323.166667 449.100174 11457.0 12346.75 12426.0 12527.75 12748.0
    gmfuzzer 54000 8.0 12313.500000 253.860479 11969.0 12071.50 12360.0 12501.75 12622.0
    gmfuzzer_236 54000 3.0 12335.000000 361.348862 12114.0 12126.50 12139.0 12445.50 12752.0
    gmfuzzer_honggfuzz 54000 20.0 11775.850000 398.146399 11028.0 11533.50 11807.0 12060.25 12645.0
    gmfuzzer_aflplusplus 54000 20.0 10697.450000 484.589732 9651.0 10560.00 10847.5 10969.00 11511.0
    gmfuzzer_mopt 54000 19.0 7679.052632 164.416975 7409.0 7556.50 7742.0 7791.00 7904.0
    ecofuzz 54000 20.0 7794.650000 412.444354 6621.0 7678.25 7740.0 8148.75 8260.0
    gmfuzzer_fairfuzz 54000 18.0 7359.111111 64.956899 7266.0 7297.75 7365.5 7403.75 7490.0
    gmfuzzer_ecofuzz 54000 19.0 7076.842105 166.159984 6894.0 6984.50 7051.0 7113.00 7623.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

harfbuzz_hb-shape-fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    gmfuzzer_133 9900 8.0 10258.875000 83.694064 10085.0 10237.25 10266.5 10320.50 10350.0
    gmfuzzer 9900 11.0 10264.090909 61.918421 10176.0 10224.50 10265.0 10287.00 10405.0
    gmfuzzer_236 9900 8.0 10265.625000 37.009410 10229.0 10231.75 10260.5 10287.25 10333.0
    gmfuzzer_aflplusplus 9900 20.0 10231.200000 113.447229 9967.0 10160.75 10255.0 10319.50 10375.0
    gmfuzzer_mopt 9900 20.0 9918.250000 270.109895 9244.0 9852.25 9971.5 10070.75 10236.0
    gmfuzzer_honggfuzz 9900 20.0 9780.650000 42.416823 9721.0 9746.00 9783.5 9807.50 9851.0
    gmfuzzer_fairfuzz 9900 20.0 9114.200000 413.461842 8446.0 8847.25 9137.5 9351.75 9821.0
    gmfuzzer_ecofuzz 9900 20.0 8897.900000 146.386978 8641.0 8778.50 8921.0 9034.25 9102.0
    ecofuzz 9900 20.0 8592.600000 126.409693 8430.0 8513.25 8567.5 8632.25 8889.0
    gmfuzzer_darwin 9900 2.0 8360.500000 212.839141 8210.0 8285.25 8360.5 8435.75 8511.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

jsoncpp_jsoncpp_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    gmfuzzer 82800 19.0 525.000000 0.000000 525.0 525.0 525.0 525.0 525.0
    gmfuzzer_133 82800 19.0 525.000000 0.000000 525.0 525.0 525.0 525.0 525.0
    gmfuzzer_236 82800 16.0 525.000000 0.000000 525.0 525.0 525.0 525.0 525.0
    gmfuzzer_honggfuzz 82800 19.0 524.894737 0.315302 524.0 525.0 525.0 525.0 525.0
    gmfuzzer_libfuzzer 82800 19.0 525.000000 0.000000 525.0 525.0 525.0 525.0 525.0
    gmfuzzer_darwin 82800 19.0 519.789474 0.418854 519.0 520.0 520.0 520.0 520.0
    gmfuzzer_mopt 82800 18.0 519.000000 0.000000 519.0 519.0 519.0 519.0 519.0
    gmfuzzer_fairfuzz 82800 20.0 518.500000 0.512989 518.0 518.0 518.5 519.0 519.0
    darwin 82800 19.0 517.631579 2.947891 506.0 518.0 518.0 519.0 520.0
    gmfuzzer_ecofuzz 82800 18.0 518.444444 0.511310 518.0 518.0 518.0 519.0 519.0
    ecofuzz 82800 17.0 515.000000 6.819091 498.0 517.0 517.0 519.0 520.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

lcms_cms_transform_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    gmfuzzer 82800 18.0 2047.444444 124.098264 1702.0 1970.00 2084.5 2148.5 2176.0
    gmfuzzer_236 82800 19.0 2040.052632 135.160593 1670.0 1964.00 2034.0 2138.5 2265.0
    gmfuzzer_133 82800 18.0 2057.388889 94.843611 1908.0 1983.50 2027.0 2133.5 2211.0
    gmfuzzer_libfuzzer 82800 20.0 1998.500000 103.741835 1688.0 1948.75 1993.5 2073.0 2128.0
    gmfuzzer_honggfuzz 82800 17.0 1973.764706 117.695332 1672.0 1919.00 1949.0 2045.0 2161.0
    gmfuzzer_aflplusplus 82800 19.0 1924.052632 105.830826 1690.0 1868.50 1925.0 1987.0 2082.0
    pastis 82800 7.0 1910.000000 198.308682 1553.0 1829.50 1912.0 2056.0 2134.0
    gmfuzzer_darwin 82800 19.0 1387.578947 407.806915 679.0 962.50 1618.0 1679.5 1760.0
    darwin 82800 19.0 1359.315789 450.467171 652.0 844.00 1596.0 1710.0 1779.0
    gmfuzzer_fairfuzz 82800 4.0 1090.500000 480.713706 678.0 681.00 1042.5 1452.0 1599.0
    gmfuzzer_ecofuzz 82800 1.0 918.000000 NaN 918.0 918.00 918.0 918.0 918.0
    gmfuzzer_mopt 82800 7.0 1003.000000 403.844442 653.0 683.00 903.0 1249.5 1600.0
    ecofuzz 82800 17.0 1069.294118 448.808250 641.0 649.00 874.0 1543.0 1734.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libjpeg-turbo_libjpeg_turbo_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    gmfuzzer 82800 19.0 3088.210526 1.228321 3085.0 3088.0 3089.0 3089.0 3089.0
    gmfuzzer_libfuzzer 82800 17.0 3087.529412 2.153998 3083.0 3087.0 3089.0 3089.0 3089.0
    gmfuzzer_133 82800 20.0 3088.050000 1.503505 3084.0 3088.0 3088.5 3089.0 3089.0
    gmfuzzer_236 82800 18.0 3087.555556 1.976992 3083.0 3087.0 3088.5 3089.0 3089.0
    gmfuzzer_aflplusplus 82800 20.0 3079.300000 10.770818 3036.0 3081.0 3082.0 3083.0 3088.0
    gmfuzzer_honggfuzz 82800 19.0 3081.315789 2.688507 3075.0 3081.0 3081.0 3082.5 3086.0
    ecofuzz 82800 19.0 2977.421053 51.033013 2889.0 2942.0 2980.0 3017.0 3063.0
    gmfuzzer_fairfuzz 82800 3.0 2982.666667 16.862186 2971.0 2973.0 2975.0 2988.5 3002.0
    gmfuzzer_ecofuzz 82800 1.0 2767.000000 NaN 2767.0 2767.0 2767.0 2767.0 2767.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libpcap_fuzz_both summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    gmfuzzer_236 82800 20.0 3040.550000 86.085927 2866.0 2999.00 3043.0 3104.00 3200.0
    pastis 82800 17.0 2992.294118 129.227882 2716.0 2975.00 3021.0 3058.00 3198.0
    gmfuzzer 82800 19.0 2999.842105 109.167691 2830.0 2924.00 3001.0 3091.50 3166.0
    gmfuzzer_133 82800 18.0 2987.277778 88.298162 2811.0 2952.25 2979.5 3016.25 3174.0
    gmfuzzer_honggfuzz 82800 17.0 2945.235294 80.237561 2727.0 2901.00 2952.0 3012.00 3052.0
    gmfuzzer_aflplusplus 82800 18.0 2883.000000 94.388434 2675.0 2838.25 2878.5 2949.50 3011.0
    gmfuzzer_libfuzzer 82800 19.0 2651.526316 74.770292 2506.0 2600.00 2654.0 2707.00 2805.0
    gmfuzzer_ecofuzz 82800 18.0 171.888889 550.826212 34.0 42.00 43.0 43.00 2379.0
    gmfuzzer_fairfuzz 82800 19.0 41.947368 2.818070 34.0 43.00 43.0 43.00 43.0
    gmfuzzer_mopt 82800 20.0 39.950000 4.019361 33.0 40.00 42.0 42.00 43.0
    ecofuzz 82800 16.0 38.437500 4.718315 33.0 34.00 38.5 43.00 43.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libpng_libpng_read_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    gmfuzzer_libfuzzer 82800 18.0 2025.944444 0.802366 2024.0 2026.0 2026.0 2026.00 2027.0
    gmfuzzer_aflplusplus 82800 19.0 2047.947368 33.295768 2017.0 2019.0 2020.0 2072.50 2101.0
    gmfuzzer 82800 18.0 2038.833333 32.740064 2018.0 2019.0 2019.0 2056.50 2101.0
    gmfuzzer_133 82800 17.0 2034.058824 32.036055 2018.0 2018.0 2019.0 2025.00 2101.0
    gmfuzzer_236 82800 17.0 2033.294118 32.157357 2017.0 2019.0 2019.0 2020.00 2101.0
    gmfuzzer_honggfuzz 82800 20.0 2019.700000 2.830287 2018.0 2018.0 2018.0 2019.25 2026.0
    gmfuzzer_darwin 82800 20.0 2009.600000 9.167448 1991.0 2009.0 2013.5 2016.00 2017.0
    gmfuzzer_mopt 82800 6.0 2006.333333 17.374311 1971.0 2012.0 2012.5 2013.75 2016.0
    darwin 82800 19.0 1986.315789 24.467785 1916.0 1991.5 1994.0 1995.50 2002.0
    gmfuzzer_ecofuzz 82800 3.0 1967.333333 12.741010 1959.0 1960.0 1961.0 1971.50 1982.0
    ecofuzz 82800 18.0 1952.000000 28.383923 1863.0 1943.0 1946.0 1969.25 1998.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libxml2_xml summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    gmfuzzer_aflplusplus 41400 2.0 15443.500000 51.618795 15407.0 15425.25 15443.5 15461.75 15480.0
    gmfuzzer_darwin 41400 19.0 15324.263158 114.906649 15102.0 15259.00 15327.0 15405.50 15510.0
    gmfuzzer_honggfuzz 41400 20.0 15285.850000 64.999008 15192.0 15242.50 15274.5 15315.75 15418.0
    darwin 41400 20.0 15190.700000 79.745714 15007.0 15140.25 15201.5 15258.25 15309.0
    gmfuzzer_mopt 41400 13.0 15196.384615 108.333235 14963.0 15144.00 15177.0 15273.00 15346.0
    gmfuzzer_fairfuzz 41400 20.0 13671.200000 1618.460822 8556.0 13788.50 14121.5 14451.50 15194.0
    gmfuzzer_ecofuzz 41400 20.0 13419.350000 245.334846 12965.0 13292.50 13479.0 13578.50 13828.0
    ecofuzz 41400 16.0 11822.187500 161.607021 11461.0 11702.75 11813.0 11946.50 12144.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libxslt_xpath summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    gmfuzzer_236 82800 20.0 11306.100000 39.015382 11209.0 11283.75 11306.0 11329.50 11365.0
    gmfuzzer 82800 19.0 11295.473684 51.573430 11206.0 11268.00 11288.0 11323.50 11392.0
    gmfuzzer_133 82800 19.0 11280.789474 45.085818 11195.0 11265.50 11281.0 11302.50 11384.0
    gmfuzzer_honggfuzz 82800 19.0 11087.368421 57.450665 10937.0 11078.50 11103.0 11124.50 11153.0
    gmfuzzer_fairfuzz 82800 19.0 11028.631579 64.893255 10928.0 10981.50 11022.0 11058.00 11150.0
    gmfuzzer_darwin 82800 13.0 10993.615385 42.390129 10918.0 10962.00 11000.0 11021.00 11058.0
    gmfuzzer_mopt 82800 20.0 10805.850000 37.374738 10742.0 10781.00 10805.5 10826.25 10882.0
    darwin 82800 18.0 10701.833333 151.503155 10211.0 10690.00 10741.0 10789.00 10838.0
    gmfuzzer_ecofuzz 82800 18.0 9737.111111 585.466620 8957.0 9035.75 10130.5 10175.00 10415.0
    ecofuzz 82800 16.0 9754.375000 590.450210 8961.0 9121.25 9847.5 10343.00 10407.0
    pastis 82800 18.0 5971.611111 4.473962 5967.0 5969.00 5970.5 5973.00 5982.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

mbedtls_fuzz_dtlsclient summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    gmfuzzer_fairfuzz 82800 18.0 2793.111111 22.180778 2746.0 2781.00 2793.0 2810.50 2826.0
    gmfuzzer_aflplusplus 82800 18.0 2787.611111 24.056646 2724.0 2779.00 2789.5 2801.00 2828.0
    gmfuzzer_236 82800 20.0 2746.400000 25.750626 2703.0 2722.25 2756.0 2766.50 2781.0
    gmfuzzer_133 82800 19.0 2749.631579 24.757065 2705.0 2732.00 2748.0 2767.50 2795.0
    gmfuzzer_libfuzzer 82800 18.0 2747.333333 110.618050 2702.0 2711.00 2719.5 2724.75 3185.0
    gmfuzzer 82800 20.0 2719.650000 27.413548 2667.0 2699.75 2716.5 2729.25 2771.0
    gmfuzzer_mopt 82800 17.0 2712.529412 20.667358 2690.0 2699.00 2709.0 2715.00 2763.0
    gmfuzzer_ecofuzz 82800 18.0 2688.888889 17.333710 2645.0 2682.25 2693.5 2700.50 2713.0
    ecofuzz 82800 19.0 2665.368421 81.621219 2448.0 2673.50 2692.0 2706.50 2755.0
    gmfuzzer_honggfuzz 82800 19.0 2601.578947 25.801197 2559.0 2586.50 2597.0 2624.50 2644.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

openh264_decoder_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    gmfuzzer_236 36000 11.0 9574.454545 4.824182 9567.0 9571.00 9575.0 9578.00 9583.0
    gmfuzzer_133 36000 5.0 9572.800000 4.207137 9566.0 9572.00 9574.0 9575.00 9577.0
    gmfuzzer_honggfuzz 36000 20.0 9567.750000 6.750244 9552.0 9562.00 9570.0 9572.25 9577.0
    gmfuzzer 36000 1.0 9564.000000 NaN 9564.0 9564.00 9564.0 9564.00 9564.0
    gmfuzzer_aflplusplus 36000 20.0 9507.400000 42.931279 9388.0 9491.25 9516.0 9535.25 9556.0
    gmfuzzer_mopt 36000 13.0 9350.153846 113.882137 9083.0 9310.00 9342.0 9445.00 9498.0
    gmfuzzer_ecofuzz 36000 20.0 9092.800000 82.960739 8856.0 9047.50 9112.5 9156.50 9188.0
    gmfuzzer_fairfuzz 36000 20.0 8943.450000 311.788639 8169.0 8779.25 8965.5 9201.25 9297.0
    ecofuzz 36000 20.0 8701.050000 80.620929 8573.0 8650.00 8705.0 8762.75 8826.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

openssl_x509 summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    gmfuzzer_236 82800 19.0 5832.842105 1.014515 5832.0 5832.0 5833.0 5833.0 5835.0
    gmfuzzer 82800 19.0 5832.526316 1.073334 5830.0 5832.0 5832.0 5833.0 5835.0
    gmfuzzer_133 82800 20.0 5832.850000 1.225819 5832.0 5832.0 5832.0 5834.0 5836.0
    gmfuzzer_darwin 82800 19.0 5833.000000 1.527525 5830.0 5832.0 5832.0 5834.0 5836.0
    gmfuzzer_fairfuzz 82800 16.0 5832.125000 1.258306 5830.0 5831.0 5832.0 5832.5 5834.0
    gmfuzzer_mopt 82800 18.0 5831.611111 1.460817 5828.0 5831.0 5831.5 5832.0 5834.0
    gmfuzzer_libfuzzer 82800 18.0 5828.500000 4.949747 5818.0 5828.0 5830.5 5832.0 5834.0
    darwin 82800 19.0 5825.736842 4.919825 5816.0 5822.0 5827.0 5830.0 5831.0
    gmfuzzer_ecofuzz 82800 19.0 5821.157895 5.500133 5810.0 5818.5 5824.0 5824.5 5828.0
    ecofuzz 82800 19.0 5810.052632 8.714443 5796.0 5804.5 5808.0 5815.5 5824.0
    gmfuzzer_honggfuzz 82800 16.0 5795.875000 3.500000 5790.0 5793.0 5796.0 5798.5 5801.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

openthread_ot-ip6-send-fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    gmfuzzer_236 82800 19.0 3483.315789 396.935910 3044.0 3075.50 3478.0 3670.00 4338.0
    gmfuzzer_aflplusplus 82800 19.0 3381.052632 195.160468 3038.0 3265.50 3474.0 3506.50 3586.0
    gmfuzzer 82800 18.0 3456.333333 305.395019 3078.0 3216.75 3459.0 3507.75 4257.0
    gmfuzzer_133 82800 20.0 3432.150000 370.867869 3046.0 3080.50 3427.0 3636.75 4156.0
    gmfuzzer_libfuzzer 82800 19.0 3112.000000 83.189209 3059.0 3083.00 3085.0 3091.00 3355.0
    gmfuzzer_honggfuzz 82800 20.0 3150.450000 189.580833 2920.0 3050.50 3076.5 3182.25 3749.0
    pastis 82800 17.0 3151.176471 179.349601 3045.0 3058.00 3074.0 3120.00 3579.0
    gmfuzzer_fairfuzz 82800 19.0 2930.368421 80.289207 2811.0 2891.50 2895.0 3029.00 3049.0
    gmfuzzer_mopt 82800 19.0 2925.105263 121.851866 2809.0 2862.00 2893.0 2958.50 3322.0
    ecofuzz 82800 18.0 2888.555556 68.860663 2813.0 2827.00 2892.5 2912.00 3054.0
    gmfuzzer_ecofuzz 82800 20.0 2849.150000 46.565489 2802.0 2809.25 2828.5 2893.00 2919.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

proj4_proj_crs_to_crs_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    gmfuzzer_236 31500 3.0 7162.333333 101.041246 7049.0 7122.00 7195.0 7219.00 7243.0
    gmfuzzer 31500 2.0 7144.000000 94.752309 7077.0 7110.50 7144.0 7177.50 7211.0
    gmfuzzer_133 31500 6.0 7149.333333 86.797849 7031.0 7104.50 7134.0 7210.75 7266.0
    gmfuzzer_honggfuzz 31500 20.0 6986.600000 203.921399 6471.0 6893.50 6976.0 7113.25 7363.0
    gmfuzzer_libfuzzer 31500 1.0 6640.000000 NaN 6640.0 6640.00 6640.0 6640.00 6640.0
    gmfuzzer_aflplusplus 31500 20.0 6601.850000 115.008592 6202.0 6570.50 6615.0 6670.25 6731.0
    gmfuzzer_fairfuzz 31500 20.0 815.650000 6.351751 804.0 808.75 818.5 819.00 829.0
    gmfuzzer_ecofuzz 31500 20.0 756.750000 70.110345 497.0 742.50 745.0 804.75 818.0
    gmfuzzer_mopt 31500 20.0 766.150000 37.502667 732.0 732.00 740.5 805.25 820.0
    ecofuzz 31500 20.0 706.150000 119.848709 470.0 720.00 733.5 790.50 817.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

re2_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    gmfuzzer_236 82800 19.0 2885.315789 2.404917 2882.0 2883.00 2885.0 2887.00 2890.0
    gmfuzzer_libfuzzer 82800 19.0 2884.947368 1.580214 2882.0 2884.00 2885.0 2885.50 2888.0
    gmfuzzer 82800 18.0 2884.277778 1.487420 2882.0 2883.00 2884.0 2885.75 2887.0
    gmfuzzer_133 82800 20.0 2884.250000 2.468219 2880.0 2882.75 2884.0 2886.00 2889.0
    gmfuzzer_darwin 82800 19.0 2882.684211 3.249831 2878.0 2880.50 2883.0 2885.00 2889.0
    gmfuzzer_mopt 82800 3.0 2880.333333 3.055050 2877.0 2879.00 2881.0 2882.00 2883.0
    gmfuzzer_honggfuzz 82800 19.0 2868.631579 5.283207 2862.0 2865.00 2868.0 2870.00 2880.0
    darwin 82800 18.0 2865.611111 5.637468 2854.0 2862.00 2867.0 2870.00 2872.0
    pastis 82800 18.0 2862.833333 4.655800 2853.0 2860.50 2863.0 2864.00 2874.0
    ecofuzz 82800 17.0 2832.764706 47.852024 2684.0 2837.00 2846.0 2859.00 2871.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

sqlite3_ossfuzz summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    darwin 80100 18.0 18759.722222 338.358634 18281.0 18499.75 18789.5 18949.75 19326.0
    gmfuzzer_aflplusplus 80100 11.0 14563.818182 362.417389 13934.0 14467.00 14608.0 14683.50 15326.0
    gmfuzzer_mopt 80100 11.0 13976.272727 314.301158 13527.0 13701.50 14036.0 14171.50 14524.0
    gmfuzzer_darwin 80100 19.0 13923.421053 397.044262 12989.0 13653.50 13989.0 14222.50 14558.0
    gmfuzzer_honggfuzz 80100 17.0 13716.882353 330.354522 12775.0 13657.00 13754.0 13866.00 14309.0
    ecofuzz 80100 17.0 12577.176471 375.025205 12070.0 12356.00 12467.0 12647.00 13612.0
    gmfuzzer_fairfuzz 80100 18.0 12076.666667 392.988100 11103.0 11891.00 12185.0 12347.00 12631.0
    gmfuzzer_ecofuzz 80100 20.0 10815.750000 131.469778 10658.0 10722.50 10800.5 10868.25 11176.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

stb_stbi_read_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    gmfuzzer 82800 19.0 2174.578947 47.599388 2126.0 2129.00 2166.0 2202.00 2277.0
    gmfuzzer_236 82800 18.0 2148.666667 36.381961 2123.0 2127.25 2130.0 2159.75 2226.0
    gmfuzzer_133 82800 20.0 2149.850000 35.167681 2123.0 2126.75 2129.0 2196.50 2209.0
    gmfuzzer_honggfuzz 82800 20.0 2149.200000 37.524168 2106.0 2123.00 2127.0 2196.50 2203.0
    gmfuzzer_aflplusplus 82800 19.0 2112.000000 30.973107 2088.0 2097.00 2101.0 2104.00 2200.0
    gmfuzzer_libfuzzer 82800 17.0 2061.588235 43.198754 1987.0 2028.00 2081.0 2094.00 2120.0
    darwin 82800 20.0 2014.700000 40.783252 1965.0 1989.25 2005.0 2009.50 2093.0
    gmfuzzer_darwin 82800 18.0 1966.277778 4.253795 1954.0 1964.25 1967.5 1969.00 1972.0
    ecofuzz 82800 19.0 1949.421053 10.905227 1939.0 1943.00 1945.0 1949.50 1975.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

vorbis_decode_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    gmfuzzer_236 73800 19.0 1314.631579 39.891813 1272.0 1274.00 1328.0 1356.50 1361.0
    gmfuzzer 73800 19.0 1298.631579 33.966013 1273.0 1275.00 1279.0 1318.50 1361.0
    gmfuzzer_aflplusplus 73800 18.0 1297.000000 41.515412 1255.0 1260.25 1278.0 1343.00 1359.0
    gmfuzzer_133 73800 18.0 1296.333333 33.309599 1273.0 1275.00 1276.5 1322.50 1362.0
    gmfuzzer_libfuzzer 73800 20.0 1281.650000 17.430388 1272.0 1275.00 1276.0 1277.50 1342.0
    gmfuzzer_honggfuzz 73800 18.0 1282.833333 28.419649 1254.0 1267.25 1271.5 1282.75 1348.0
    ecofuzz 73800 18.0 1237.444444 19.424733 1177.0 1242.00 1245.0 1247.75 1250.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

woff2_convert_woff2ttf_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    gmfuzzer_236 82800 19.0 1217.578947 10.101242 1200.0 1216.00 1221.0 1222.00 1239.0
    gmfuzzer_133 82800 19.0 1220.315789 3.145590 1215.0 1217.50 1220.0 1223.00 1226.0
    gmfuzzer 82800 18.0 1217.222222 7.158340 1200.0 1216.25 1217.5 1222.75 1225.0
    gmfuzzer_libfuzzer 82800 19.0 1215.315789 7.638009 1198.0 1214.00 1217.0 1220.00 1227.0
    gmfuzzer_aflplusplus 82800 19.0 1205.842105 3.516045 1197.0 1203.00 1207.0 1208.00 1210.0
    gmfuzzer_honggfuzz 82800 18.0 1191.055556 12.977230 1166.0 1182.25 1189.5 1199.00 1214.0
    gmfuzzer_mopt 82800 19.0 1132.789474 4.720863 1118.0 1132.00 1134.0 1135.50 1139.0
    ecofuzz 82800 18.0 1100.000000 27.608609 1017.0 1087.25 1105.0 1115.75 1134.0
    pastis 82800 20.0 770.000000 0.000000 770.0 770.00 770.0 770.00 770.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

zlib_zlib_uncompress_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    gmfuzzer_libfuzzer 82800 17.0 467.411765 4.569721 462.0 463.0 470.0 472.00 472.0
    gmfuzzer 82800 18.0 465.055556 3.947780 462.0 463.0 463.0 469.00 472.0
    gmfuzzer_133 82800 18.0 466.666667 4.588989 462.0 463.0 463.0 471.75 472.0
    gmfuzzer_236 82800 17.0 464.882353 3.497899 462.0 463.0 463.0 466.00 472.0
    pastis 82800 17.0 462.764706 3.250566 457.0 462.0 463.0 463.00 470.0
    gmfuzzer_honggfuzz 82800 17.0 462.058824 2.609372 459.0 461.0 462.0 462.00 471.0
    ecofuzz 82800 19.0 450.368421 15.225018 415.0 452.0 457.0 458.50 467.0
    gmfuzzer_aflplusplus 82800 19.0 452.736842 4.470174 447.0 449.0 452.0 458.00 459.0
    gmfuzzer_fairfuzz 82800 19.0 452.052632 4.034210 447.0 449.0 450.0 456.50 458.0
    gmfuzzer_ecofuzz 82800 19.0 450.315789 4.123815 444.0 448.5 449.0 450.50 460.0
    gmfuzzer_mopt 82800 18.0 447.888889 3.894021 444.0 444.0 447.0 449.75 456.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

experiment data

You can download the raw data for this report here.

Check out the documentation on how to create customized reports using this data. Also see some example Colab notebooks for doing custom analysis on the data here.

Experiment Description:

(None,)