FuzzBench: 2023-09-27-orchestra-1 report

(experiment incomplete/still running...)

experiment summary

We show two different aggregate (cross-benchmark) rankings of fuzzers. The first is based on the average of per-benchmarks scores, where the score represents the percentage of the highest reached median code-coverage on a given benchmark (higher value is better). The second ranking shows the average rank of fuzzers, after we rank them on each benchmark according to their median reached code-covereges (lower value is better).
By avg. score
average normalized score
fuzzer
orchestra_339 99.66
orchestra_4310 99.49
orchestra_236 99.36
orchestra_216 94.86
orchestra_116 86.10
By avg. rank
average rank
fuzzer
orchestra_339 2.22
orchestra_236 2.26
orchestra_4310 2.57
orchestra_216 3.04
orchestra_116 3.52
  • Critical difference diagram
    The diagram visualizes the average rank of fuzzers (second ranking above) while showing the significance of the differences as well. What is considered a "critical difference" (CD) is based on the Friedman/Nemenyi post-hoc test. See more in the documentation.
    Note: If a fuzzer does not support all benchmarks, its ranking as shown in this diagram can be lower than it should be. So please check the list of supported benchmarks for the fuzzer(s) of your interest. The list could be specified in the fuzzer's README.md like this.
  • Median relative code-coverages on each benchmark

    Note: The relative coverage summary table shows the median relative performance of each fuzzer to the experiment maximum. Thus the highest relative performance may not be 100%.
    trial_relative_coverage = trial_coverage / experiment_max_coverage

      orchestra_339 orchestra_4310 orchestra_116 orchestra_236 orchestra_216
    FuzzerMedian 96.00 96.00 96.50 95.00 95.50
    FuzzerMean 94.91 94.74 94.65 94.48 94.45
    bloaty_fuzz_target 89.00 87.00 nan 87.00 88.00
    curl_curl_fuzzer_http 94.00 94.00 94.00 95.00 nan
    freetype2_ftfuzzer 94.00 93.00 91.00 92.00 89.00
    harfbuzz_hb-shape-fuzzer 98.00 98.00 97.00 97.00 98.00
    jsoncpp_jsoncpp_fuzzer 100.00 100.00 100.00 100.00 100.00
    lcms_cms_transform_fuzzer 91.00 91.00 nan 91.00 91.00
    libjpeg-turbo_libjpeg_turbo_fuzzer 96.00 96.00 97.00 97.00 97.00
    libpcap_fuzz_both 93.00 93.00 92.00 93.00 93.00
    libpng_libpng_read_fuzzer 99.00 99.00 nan 99.00 99.00
    libxml2_xml 95.00 96.00 97.00 93.00 95.00
    libxslt_xpath 95.00 95.00 95.00 95.00 95.00
    mbedtls_fuzz_dtlsclient 94.00 92.00 95.00 92.00 91.00
    openh264_decoder_fuzzer 99.00 99.00 99.00 99.00 99.00
    openssl_x509 99.00 99.00 99.00 99.00 99.00
    openthread_ot-ip6-send-fuzzer 86.00 86.00 85.00 85.00 85.00
    proj4_proj_crs_to_crs_fuzzer 96.00 96.00 92.00 94.00 95.00
    re2_fuzzer 99.00 99.00 99.00 99.00 99.00
    sqlite3_ossfuzz 86.00 86.00 84.00 86.00 86.00
    stb_stbi_read_fuzzer 96.00 96.00 96.00 96.00 96.00
    systemd_fuzz-link-parser 88.00 88.00 85.00 88.00 88.00
    vorbis_decode_fuzzer 98.00 98.00 98.00 98.00 98.00
    woff2_convert_woff2ttf_fuzzer 99.00 99.00 99.00 99.00 98.00
    zlib_zlib_uncompress_fuzzer 99.00 99.00 99.00 99.00 99.00
    • Fuzzers are sorted by "FuzzerMean" (average median relative coverage), highest on the left.
    • Green background = highest relative median coverage.
    • Blue gradient background = greater than 95% relative median coverage.

bloaty_fuzz_target summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    orchestra_339 82800 13.0 5401.846154 131.812269 5116.0 5343.00 5411.0 5461.00 5608.0
    orchestra_216 82800 14.0 5383.428571 203.596023 4980.0 5281.75 5358.5 5465.75 5770.0
    orchestra_4310 82800 13.0 5343.461538 388.691312 4577.0 5196.00 5318.0 5599.00 6052.0
    orchestra_236 82800 14.0 5290.285714 215.826294 4829.0 5183.75 5316.0 5418.00 5628.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

curl_curl_fuzzer_http summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    orchestra_236 82800 20.0 9887.30 303.608838 9225.0 9726.75 9948.0 10037.50 10420.0
    orchestra_339 82800 20.0 9888.60 173.931142 9626.0 9785.75 9889.0 9949.00 10342.0
    orchestra_4310 82800 20.0 9897.80 259.274413 9091.0 9763.75 9889.0 10090.00 10219.0
    orchestra_116 82800 20.0 9793.85 281.767032 9143.0 9684.00 9808.5 10025.25 10236.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

freetype2_ftfuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    orchestra_339 82800 20.0 11849.95 682.416255 10578.0 11171.00 12164.5 12401.00 12712.0
    orchestra_4310 82800 20.0 11798.60 641.942726 10906.0 11218.25 11984.0 12248.25 12804.0
    orchestra_236 82800 20.0 11763.00 655.688710 10794.0 11115.00 11933.5 12274.75 12856.0
    orchestra_116 82800 20.0 11498.50 757.346580 10068.0 10891.25 11752.5 11950.75 12808.0
    orchestra_216 82800 20.0 11519.05 875.713752 10035.0 10791.50 11525.5 12332.25 12679.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

harfbuzz_hb-shape-fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    orchestra_339 82800 20.0 10000.60 91.218304 9842.0 9936.50 10023.0 10083.50 10130.0
    orchestra_216 82800 20.0 9952.65 162.329190 9598.0 9836.25 9993.0 10089.50 10191.0
    orchestra_4310 82800 20.0 9977.35 132.036787 9623.0 9925.00 9991.5 10067.00 10172.0
    orchestra_236 82800 20.0 9932.10 61.094319 9803.0 9907.25 9941.0 9981.75 10014.0
    orchestra_116 82800 20.0 9926.65 71.561733 9793.0 9869.75 9930.0 9970.50 10069.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

jsoncpp_jsoncpp_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    orchestra_116 82800 20.0 525.0 0.0 525.0 525.0 525.0 525.0 525.0
    orchestra_216 82800 20.0 525.0 0.0 525.0 525.0 525.0 525.0 525.0
    orchestra_236 82800 20.0 525.0 0.0 525.0 525.0 525.0 525.0 525.0
    orchestra_339 82800 20.0 525.0 0.0 525.0 525.0 525.0 525.0 525.0
    orchestra_4310 82800 20.0 525.0 0.0 525.0 525.0 525.0 525.0 525.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

lcms_cms_transform_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    orchestra_236 82800 20.0 2070.05 72.354737 1976.0 2009.50 2052.5 2131.25 2221.0
    orchestra_216 82800 20.0 2058.65 71.278014 1959.0 2000.00 2052.0 2102.00 2233.0
    orchestra_339 82800 20.0 2054.55 80.628110 1932.0 2003.25 2049.5 2124.75 2219.0
    orchestra_4310 82800 20.0 2056.15 80.895173 1936.0 1989.50 2048.0 2129.00 2179.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libjpeg-turbo_libjpeg_turbo_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    orchestra_216 82800 20.0 2998.250000 59.102832 2851.0 2984.5 3008.0 3027.75 3075.0
    orchestra_236 82800 20.0 2993.950000 57.764016 2901.0 2955.5 3007.0 3049.25 3076.0
    orchestra_116 82800 19.0 2986.157895 57.752598 2880.0 2951.5 3004.0 3024.50 3074.0
    orchestra_339 82800 19.0 2978.631579 69.287814 2853.0 2933.0 2973.0 3020.50 3075.0
    orchestra_4310 82800 20.0 2981.900000 59.157150 2841.0 2954.5 2972.0 3013.00 3081.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libpcap_fuzz_both summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    orchestra_236 82800 20.0 2818.25 73.719151 2660.0 2772.25 2824.5 2853.00 2982.0
    orchestra_339 82800 20.0 2811.80 89.876875 2649.0 2745.50 2817.0 2870.50 2936.0
    orchestra_4310 82800 20.0 2830.05 63.781267 2739.0 2780.75 2809.5 2888.25 2963.0
    orchestra_216 82800 20.0 2808.45 70.811443 2647.0 2773.75 2805.5 2818.75 2961.0
    orchestra_116 82800 20.0 2801.10 106.504213 2633.0 2723.00 2778.0 2875.75 3007.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libpng_libpng_read_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    orchestra_216 82800 19.0 2017.368421 0.760886 2016.0 2017.0 2018.0 2018.0 2018.0
    orchestra_236 82800 19.0 2017.789474 0.630604 2017.0 2017.0 2018.0 2018.0 2019.0
    orchestra_339 82800 19.0 2018.105263 1.328940 2016.0 2018.0 2018.0 2018.0 2023.0
    orchestra_4310 82800 20.0 2017.200000 1.005249 2014.0 2017.0 2017.0 2018.0 2018.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libxml2_xml summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    orchestra_116 82800 20.0 14061.05 408.690524 13254.0 13681.50 14210.5 14389.00 14595.0
    orchestra_4310 82800 20.0 13952.05 458.444335 13192.0 13572.25 14061.5 14319.75 14626.0
    orchestra_339 82800 20.0 13948.25 379.268947 13397.0 13604.50 14020.0 14192.50 14532.0
    orchestra_216 82800 20.0 13957.35 408.719602 13310.0 13670.00 13985.0 14263.25 14579.0
    orchestra_236 82800 20.0 13860.65 418.158906 13348.0 13516.25 13740.0 14243.75 14571.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libxslt_xpath summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    orchestra_339 82800 20.0 9719.60 132.284144 9424.0 9632.50 9697.5 9787.00 9951.0
    orchestra_116 82800 20.0 9701.50 130.109468 9490.0 9636.50 9691.5 9729.00 10114.0
    orchestra_236 82800 20.0 9663.05 97.409324 9479.0 9587.50 9681.0 9737.75 9816.0
    orchestra_216 82800 20.0 9652.55 90.799649 9525.0 9567.75 9651.0 9700.50 9828.0
    orchestra_4310 82800 20.0 9666.00 101.658872 9483.0 9609.00 9650.5 9708.25 9890.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

mbedtls_fuzz_dtlsclient summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    orchestra_116 82800 20.0 2562.25 104.142046 2308.0 2537.75 2604.5 2625.00 2689.0
    orchestra_339 82800 20.0 2564.50 79.033970 2423.0 2498.50 2568.5 2617.75 2703.0
    orchestra_4310 82800 20.0 2488.90 131.323586 2240.0 2375.75 2530.0 2598.50 2666.0
    orchestra_236 82800 20.0 2501.15 146.755176 2128.0 2409.75 2517.0 2615.00 2721.0
    orchestra_216 82800 20.0 2495.65 81.178345 2375.0 2415.75 2495.0 2567.25 2612.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

openh264_decoder_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    orchestra_4310 82800 20.0 9513.75 20.439577 9475.0 9501.75 9514.5 9530.00 9544.0
    orchestra_339 82800 20.0 9510.50 29.491301 9469.0 9483.75 9507.5 9527.25 9568.0
    orchestra_236 82800 20.0 9500.65 20.845989 9458.0 9491.00 9501.5 9511.00 9545.0
    orchestra_216 82800 20.0 9503.25 25.945033 9466.0 9486.50 9499.5 9518.25 9567.0
    orchestra_116 82800 20.0 9490.60 22.286177 9462.0 9474.50 9486.0 9508.00 9550.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

openssl_x509 summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    orchestra_236 82800 20.0 5806.550000 8.708586 5788.0 5799.25 5810.0 5813.25 5817.0
    orchestra_216 82800 20.0 5806.350000 10.864113 5778.0 5803.25 5809.5 5812.25 5825.0
    orchestra_116 82800 20.0 5805.750000 10.135269 5787.0 5797.50 5806.5 5815.25 5819.0
    orchestra_4310 82800 19.0 5801.578947 8.016066 5786.0 5797.50 5804.0 5808.50 5814.0
    orchestra_339 82800 19.0 5800.526316 11.635045 5778.0 5791.00 5799.0 5810.50 5819.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

openthread_ot-ip6-send-fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    orchestra_4310 82800 20.0 3154.70 163.754662 3050.0 3057.75 3067.0 3165.25 3521.0
    orchestra_339 82800 20.0 3080.30 74.298190 3051.0 3056.75 3064.0 3072.25 3394.0
    orchestra_116 82800 20.0 3089.80 130.181088 3030.0 3041.00 3047.0 3060.00 3561.0
    orchestra_236 82800 20.0 3080.45 110.711324 3032.0 3040.75 3045.0 3054.00 3411.0
    orchestra_216 82800 20.0 3111.15 139.461096 3033.0 3039.75 3042.0 3062.25 3414.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

proj4_proj_crs_to_crs_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    orchestra_4310 82800 20.0 6905.20 128.911475 6663.0 6825.25 6909.0 6986.25 7128.0
    orchestra_339 82800 20.0 6834.55 124.809529 6569.0 6771.50 6843.5 6921.00 7044.0
    orchestra_216 82800 20.0 6800.05 130.962620 6504.0 6734.25 6794.5 6889.00 7047.0
    orchestra_236 82800 20.0 6737.55 112.023247 6526.0 6680.75 6752.5 6803.25 6963.0
    orchestra_116 82800 20.0 6586.90 126.988976 6357.0 6502.75 6590.0 6689.25 6816.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

re2_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    orchestra_4310 82800 20.0 2877.45 4.135533 2864.0 2875.75 2879.0 2880.00 2882.0
    orchestra_339 82800 20.0 2877.95 3.734265 2867.0 2876.75 2878.0 2881.00 2882.0
    orchestra_216 82800 20.0 2874.10 5.802903 2863.0 2871.25 2876.0 2878.25 2881.0
    orchestra_236 82800 20.0 2873.65 5.967235 2863.0 2868.25 2876.0 2878.00 2882.0
    orchestra_116 82800 20.0 2871.50 6.427736 2862.0 2864.75 2872.5 2877.00 2881.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

sqlite3_ossfuzz summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    orchestra_4310 82800 20.0 11969.300000 276.018897 11520.0 11803.00 11909.0 12117.50 12836.0
    orchestra_236 82800 20.0 12022.050000 494.641335 11607.0 11768.25 11900.0 11968.75 13742.0
    orchestra_216 82800 19.0 11930.947368 414.308188 11421.0 11739.00 11864.0 12000.50 13236.0
    orchestra_339 82800 20.0 11941.200000 336.057890 11633.0 11792.75 11846.0 11993.25 13251.0
    orchestra_116 82800 20.0 11805.600000 488.965116 11260.0 11533.25 11654.5 11873.00 13309.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

stb_stbi_read_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    orchestra_236 82800 20.0 2091.35 14.368368 2050.0 2083.5 2095.0 2102.00 2107.0
    orchestra_216 82800 20.0 2095.90 17.432275 2079.0 2086.0 2092.5 2098.00 2162.0
    orchestra_116 82800 20.0 2090.35 12.478508 2054.0 2084.5 2091.5 2100.25 2108.0
    orchestra_339 82800 20.0 2081.90 18.842561 2019.0 2077.5 2087.5 2093.00 2104.0
    orchestra_4310 82800 20.0 2087.60 10.049352 2064.0 2083.0 2087.0 2094.00 2108.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

vorbis_decode_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    orchestra_236 82800 20.0 1341.90 41.534893 1177.0 1345.00 1354.0 1360.25 1362.0
    orchestra_339 82800 20.0 1335.35 44.091144 1202.0 1326.75 1352.0 1360.25 1366.0
    orchestra_4310 82800 20.0 1346.05 18.285672 1304.0 1333.50 1351.5 1359.25 1369.0
    orchestra_116 82800 20.0 1347.50 10.733617 1324.0 1342.25 1349.0 1356.00 1361.0
    orchestra_216 82800 20.0 1340.55 23.013669 1281.0 1330.50 1346.0 1357.50 1366.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

woff2_convert_woff2ttf_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    orchestra_116 82800 20.0 1213.30 5.037752 1198.0 1212.00 1213.0 1214.25 1224.0
    orchestra_236 82800 20.0 1212.05 5.443248 1200.0 1210.75 1213.0 1214.00 1222.0
    orchestra_4310 82800 20.0 1210.45 7.315988 1196.0 1202.75 1212.5 1214.00 1223.0
    orchestra_339 82800 20.0 1212.00 6.232343 1200.0 1208.50 1212.0 1215.50 1224.0
    orchestra_216 82800 20.0 1209.30 7.476982 1195.0 1204.00 1209.0 1213.00 1224.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

zlib_zlib_uncompress_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    orchestra_116 82800 20.0 471.30 1.559352 465.0 471.00 472.0 472.0 472.0
    orchestra_216 82800 20.0 471.90 0.447214 471.0 472.00 472.0 472.0 473.0
    orchestra_236 82800 20.0 471.75 0.444262 471.0 471.75 472.0 472.0 472.0
    orchestra_339 82800 20.0 471.95 0.394034 471.0 472.00 472.0 472.0 473.0
    orchestra_4310 82800 20.0 471.35 2.007224 463.0 471.75 472.0 472.0 472.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

experiment data

You can download the raw data for this report here.

Check out the documentation on how to create customized reports using this data. Also see some example Colab notebooks for doing custom analysis on the data here.

Experiment Description:

(None,)