FuzzBench: 2023-04-27-symcts report

(experiment incomplete/still running...)

experiment summary

We show two different aggregate (cross-benchmark) rankings of fuzzers. The first is based on the average of per-benchmarks scores, where the score represents the percentage of the highest reached median code-coverage on a given benchmark (higher value is better). The second ranking shows the average rank of fuzzers, after we rank them on each benchmark according to their median reached code-covereges (lower value is better).
By avg. score
average normalized score
fuzzer
symcts_symqemu_afl 88.38
aflplusplus 88.12
symcts_afl 78.22
libfuzzer 77.71
symcc_aflplusplus 77.23
honggfuzz 74.95
centipede 47.38
symsan 40.03
symcts 33.24
symcts_symqemu 32.90
By avg. rank
average rank
fuzzer
symcts_symqemu_afl 3.00
symcts_afl 3.10
symcc_aflplusplus 4.05
aflplusplus 4.10
libfuzzer 4.86
honggfuzz 4.90
symcts 6.81
centipede 6.90
symsan 7.19
symcts_symqemu 7.38
  • Critical difference diagram
    The diagram visualizes the average rank of fuzzers (second ranking above) while showing the significance of the differences as well. What is considered a "critical difference" (CD) is based on the Friedman/Nemenyi post-hoc test. See more in the documentation.
    Note: If a fuzzer does not support all benchmarks, its ranking as shown in this diagram can be lower than it should be. So please check the list of supported benchmarks for the fuzzer(s) of your interest. The list could be specified in the fuzzer's README.md like this.
  • Median relative code-coverages on each benchmark

    Note: The relative coverage summary table shows the median relative performance of each fuzzer to the experiment maximum. Thus the highest relative performance may not be 100%.
    trial_relative_coverage = trial_coverage / experiment_max_coverage

      symcts_afl symcts_symqemu_afl symcc_aflplusplus aflplusplus honggfuzz libfuzzer symcts symsan symcts_symqemu centipede
    FuzzerMedian 91.00 89.00 87.50 86.00 78.00 75.00 76.00 73.00 63.00 49.00
    FuzzerMean 89.12 84.70 83.06 81.14 73.45 71.86 63.40 62.42 50.75 48.84
    bloaty_fuzz_target 72.00 71.00 36.00 36.00 35.00 29.00 nan 9.00 71.00 16.00
    curl_curl_fuzzer_http 91.00 88.00 86.00 87.00 95.00 78.00 nan 77.00 13.00 nan
    freetype2_ftfuzzer 90.00 90.00 89.00 87.00 73.00 43.00 nan nan 36.00 22.00
    harfbuzz_hb-shape-fuzzer 97.00 94.00 93.00 94.00 83.00 67.00 nan nan nan 49.00
    jsoncpp_jsoncpp_fuzzer 99.00 99.00 99.00 99.00 99.00 100.00 nan nan 72.00 99.00
    lcms_cms_transform_fuzzer 91.00 90.00 nan 81.00 71.00 75.00 89.00 31.00 84.00 1.00
    libjpeg-turbo_libjpeg_turbo_fuzzer 97.00 41.00 81.00 29.00 21.00 79.00 39.00 21.00 nan 24.00
    libpcap_fuzz_both 91.00 89.00 nan 94.00 88.00 81.00 nan 89.00 nan 89.00
    libpng_libpng_read_fuzzer 95.00 76.00 90.00 84.00 90.00 96.00 90.00 73.00 nan 0.00
    libxml2_xml nan nan 97.00 96.00 63.00 65.00 nan nan nan 52.00
    libxslt_xpath nan 92.00 89.00 88.00 72.00 72.00 nan 88.00 nan 62.00
    openh264_decoder_fuzzer nan 99.00 98.00 99.00 97.00 97.00 9.00 nan 8.00 96.00
    openssl_x509 71.00 71.00 71.00 71.00 91.00 71.00 70.00 71.00 55.00 71.00
    openthread_ot-ip6-send-fuzzer 79.00 78.00 77.00 66.00 66.00 67.00 78.00 77.00 76.00 62.00
    proj4_proj_crs_to_crs_fuzzer 83.00 87.00 90.00 86.00 97.00 90.00 nan nan nan 10.00
    re2_fuzzer 99.00 99.00 99.00 99.00 98.00 98.00 nan nan nan 96.00
    sqlite3_ossfuzz 84.00 89.00 nan 90.00 20.00 20.00 17.00 nan 18.00 49.00
    stb_stbi_read_fuzzer 92.00 85.00 85.00 76.00 63.00 28.00 74.00 nan 74.00 32.00
    systemd_fuzz-link-parser nan 73.00 73.00 73.00 nan 71.00 nan 73.00 nan nan
    vorbis_decode_fuzzer 90.00 90.00 56.00 85.00 54.00 85.00 81.00 55.00 17.00 15.00
    woff2_convert_woff2ttf_fuzzer 94.00 93.00 86.00 84.00 93.00 97.00 87.00 85.00 85.00 83.00
    • Fuzzers are sorted by "FuzzerMean" (average median relative coverage), highest on the left.
    • Green background = highest relative median coverage.
    • Blue gradient background = greater than 95% relative median coverage.

bloaty_fuzz_target summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    symcts_afl 82800 6.0 3956.500000 878.843046 2190.0 4099.75 4290.5 4413.00 4478.0
    symcts_symqemu_afl 82800 15.0 4040.200000 626.894523 1895.0 4099.50 4250.0 4311.00 4437.0
    symcts_symqemu 82800 13.0 4187.923077 164.898687 3702.0 4153.00 4249.0 4277.00 4305.0
    aflplusplus 82800 20.0 2180.250000 93.578492 1921.0 2149.50 2166.5 2209.50 2384.0
    symcc_aflplusplus 82800 18.0 2137.111111 62.265928 2037.0 2102.50 2134.5 2161.25 2292.0
    honggfuzz 82800 20.0 2110.850000 69.191782 1953.0 2068.50 2110.0 2150.50 2226.0
    libfuzzer 82800 20.0 1654.550000 279.240412 1222.0 1424.25 1739.5 1875.00 2040.0
    centipede 82800 20.0 942.300000 74.275518 827.0 889.50 961.5 997.75 1037.0
    symsan 82800 20.0 541.000000 0.000000 541.0 541.00 541.0 541.00 541.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

curl_curl_fuzzer_http summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    honggfuzz 82800 20.0 9607.650000 609.736291 8509.0 8998.75 9961.0 10054.25 10389.0
    symcts_afl 82800 1.0 9495.000000 NaN 9495.0 9495.00 9495.0 9495.00 9495.0
    symcts_symqemu_afl 82800 3.0 9418.333333 500.185299 9067.0 9132.00 9197.0 9594.00 9991.0
    aflplusplus 82800 20.0 9202.800000 438.115294 8489.0 8939.50 9136.5 9382.25 10288.0
    symcc_aflplusplus 82800 20.0 9079.450000 307.513367 8576.0 8881.75 9027.0 9324.50 9650.0
    libfuzzer 82800 20.0 8226.100000 493.885550 7026.0 7846.75 8188.5 8576.75 9082.0
    symsan 82800 1.0 8073.000000 NaN 8073.0 8073.00 8073.0 8073.00 8073.0
    symcts_symqemu 82800 20.0 1452.000000 0.000000 1452.0 1452.00 1452.0 1452.00 1452.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

freetype2_ftfuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    symcts_afl 82800 5.0 10954.40 930.109026 9350.0 11064.00 11168.0 11532.00 11658.0
    symcts_symqemu_afl 82800 5.0 10648.40 1111.448739 9269.0 9645.00 11150.0 11516.00 11662.0
    symcc_aflplusplus 82800 20.0 10805.55 871.274111 9023.0 10542.25 11089.5 11360.75 12346.0
    aflplusplus 82800 20.0 10591.65 1021.300063 9212.0 9566.25 10781.5 11375.75 12109.0
    honggfuzz 82800 20.0 8975.85 398.452173 7810.0 8805.50 9114.5 9195.50 9604.0
    libfuzzer 82800 20.0 5444.30 404.924047 4917.0 5065.00 5377.0 5794.00 6162.0
    symcts_symqemu 82800 1.0 4539.00 NaN 4539.0 4539.00 4539.0 4539.00 4539.0
    centipede 82800 19.0 2165.00 918.852545 1069.0 1186.00 2812.0 3004.00 3190.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

harfbuzz_hb-shape-fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    symcts_afl 82800 2.0 9073.50 140.714249 8974.0 9023.75 9073.5 9123.25 9173.0
    aflplusplus 82800 20.0 8884.15 214.846157 8519.0 8738.25 8829.5 9079.25 9337.0
    symcts_symqemu_afl 82800 1.0 8790.00 NaN 8790.0 8790.00 8790.0 8790.00 8790.0
    symcc_aflplusplus 82800 20.0 8767.30 154.345852 8473.0 8679.75 8736.5 8817.25 9229.0
    honggfuzz 82800 20.0 7673.75 261.212330 6749.0 7714.00 7766.0 7798.00 7860.0
    libfuzzer 82800 20.0 6340.05 187.699501 6070.0 6215.75 6300.0 6432.00 6776.0
    centipede 82800 20.0 4716.10 258.955371 4432.0 4531.75 4602.5 4970.75 5323.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

jsoncpp_jsoncpp_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libfuzzer 82800 20.0 525.000000 0.000000 525.0 525.00 525.0 525.0 525.0
    honggfuzz 82800 20.0 521.350000 2.996050 509.0 522.00 522.0 522.0 523.0
    centipede 82800 20.0 520.300000 2.657660 515.0 519.75 521.5 522.0 523.0
    aflplusplus 82800 20.0 519.800000 0.523148 518.0 520.00 520.0 520.0 520.0
    symcc_aflplusplus 82800 19.0 521.368421 2.033348 519.0 520.00 520.0 523.0 525.0
    symcts_afl 82800 7.0 519.857143 0.377964 519.0 520.00 520.0 520.0 520.0
    symcts_symqemu_afl 82800 20.0 519.850000 0.366348 519.0 520.00 520.0 520.0 520.0
    symcts_symqemu 82800 1.0 378.000000 NaN 378.0 378.00 378.0 378.0 378.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

lcms_cms_transform_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    symcts_afl 82800 20.0 2085.500000 82.345549 1948.0 2029.50 2066.5 2098.25 2260.0
    symcts_symqemu_afl 82800 20.0 2052.900000 85.240774 1889.0 1994.75 2038.0 2117.00 2223.0
    symcts 82800 20.0 2032.650000 93.524708 1836.0 1985.50 2028.0 2111.75 2174.0
    symcts_symqemu 82800 18.0 1892.388889 50.361212 1805.0 1853.50 1900.0 1935.50 1964.0
    aflplusplus 82800 20.0 1607.050000 480.395175 685.0 1625.00 1835.0 1911.00 1990.0
    libfuzzer 82800 19.0 1752.473684 91.650525 1635.0 1681.00 1716.0 1830.50 1922.0
    honggfuzz 82800 20.0 1389.600000 505.115766 705.0 737.25 1615.5 1800.50 1965.0
    symsan 82800 5.0 1100.000000 558.514995 704.0 711.00 713.0 1459.00 1913.0
    centipede 82800 20.0 36.000000 0.000000 36.0 36.00 36.0 36.00 36.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libjpeg-turbo_libjpeg_turbo_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    symcts_afl 82800 20.0 2326.450000 269.296642 1496.0 2083.75 2469.0 2513.25 2541.0
    symcc_aflplusplus 82800 20.0 1674.500000 789.953596 562.0 918.25 2082.5 2299.00 2540.0
    libfuzzer 82800 20.0 1940.100000 314.274754 743.0 1929.00 2020.5 2072.50 2251.0
    symcts_symqemu_afl 82800 15.0 1455.866667 920.186384 523.0 523.00 1058.0 2427.50 2538.0
    symcts 82800 20.0 1226.850000 357.750468 809.0 984.25 996.5 1671.00 1736.0
    aflplusplus 82800 20.0 1190.800000 800.491270 522.0 523.00 740.5 2078.25 2539.0
    centipede 82800 20.0 718.800000 202.310496 518.0 537.75 630.0 939.75 988.0
    honggfuzz 82800 20.0 728.400000 476.574222 500.0 528.25 543.0 550.75 2189.0
    symsan 82800 2.0 543.000000 28.284271 523.0 533.00 543.0 553.00 563.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libpcap_fuzz_both summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus 82800 19.0 2977.263158 102.008628 2726.0 2922.00 2990.0 3038.50 3160.0
    symcts_afl 82800 14.0 2853.500000 125.077053 2565.0 2792.00 2897.5 2927.75 3042.0
    centipede 82800 20.0 2531.900000 864.166705 101.0 2611.50 2842.0 2949.00 3141.0
    symsan 82800 4.0 2849.500000 57.396283 2792.0 2813.00 2841.0 2877.50 2924.0
    symcts_symqemu_afl 82800 20.0 2836.400000 124.575236 2619.0 2746.75 2839.5 2913.50 3105.0
    honggfuzz 82800 20.0 2790.000000 94.411195 2611.0 2733.00 2791.5 2858.75 2947.0
    libfuzzer 82800 20.0 2543.850000 131.991736 2040.0 2518.00 2561.5 2588.75 2720.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libpng_libpng_read_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libfuzzer 82800 20.0 2018.900000 17.926458 2012.0 2015.00 2015.0 2015.00 2095.0
    symcts_afl 82800 20.0 1930.950000 172.275017 1523.0 1933.75 2000.5 2017.50 2094.0
    symcc_aflplusplus 82800 20.0 1792.450000 184.624933 1516.0 1594.50 1898.5 1948.25 1995.0
    honggfuzz 82800 20.0 1791.100000 204.549029 1526.0 1580.75 1898.0 1968.25 2086.0
    symcts 82800 17.0 1865.823529 120.959309 1537.0 1827.00 1888.0 1900.00 2040.0
    aflplusplus 82800 20.0 1689.000000 392.438262 271.0 1550.75 1765.0 1961.00 1998.0
    symcts_symqemu_afl 82800 20.0 1737.450000 215.912622 1517.0 1549.75 1602.5 1948.50 2074.0
    symsan 82800 1.0 1545.000000 NaN 1545.0 1545.00 1545.0 1545.00 1545.0
    centipede 82800 20.0 7.000000 0.000000 7.0 7.00 7.0 7.00 7.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libxml2_xml summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    symcc_aflplusplus 81900 20.0 12267.65 240.538929 11607.0 12176.00 12329.5 12403.00 12659.0
    aflplusplus 81900 20.0 12194.80 307.371813 11398.0 12043.75 12254.0 12391.50 12663.0
    libfuzzer 81900 20.0 8224.15 146.739036 8057.0 8069.50 8320.0 8360.75 8375.0
    honggfuzz 81900 20.0 8073.50 321.979976 7459.0 7949.50 8052.0 8367.75 8679.0
    centipede 81900 20.0 6614.85 329.889499 5717.0 6418.75 6690.0 6865.75 7034.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libxslt_xpath summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    symcts_symqemu_afl 82800 7.0 7833.428571 125.042469 7635.0 7752.00 7883.0 7916.50 7979.0
    symcc_aflplusplus 82800 20.0 7759.550000 259.336699 7563.0 7598.25 7658.0 7800.25 8546.0
    aflplusplus 82800 20.0 7631.850000 131.488352 7297.0 7572.25 7601.5 7715.75 7903.0
    symsan 82800 2.0 7521.500000 65.760931 7475.0 7498.25 7521.5 7544.75 7568.0
    honggfuzz 82800 20.0 6186.050000 114.675503 5998.0 6150.75 6163.5 6199.50 6585.0
    libfuzzer 82800 20.0 6130.450000 71.832755 5995.0 6062.00 6161.0 6176.00 6245.0
    centipede 82800 20.0 5372.100000 66.313371 5246.0 5321.00 5379.5 5412.50 5504.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

openh264_decoder_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    symcts_symqemu_afl 82800 1.0 9565.000000 NaN 9565.0 9565.00 9565.0 9565.00 9565.0
    aflplusplus 82800 20.0 9558.650000 33.082473 9461.0 9548.50 9561.5 9567.00 9643.0
    symcc_aflplusplus 82800 19.0 9507.789474 45.954783 9401.0 9492.50 9504.0 9527.00 9613.0
    honggfuzz 82800 20.0 9438.400000 82.860949 9249.0 9392.75 9415.0 9519.25 9545.0
    libfuzzer 82800 20.0 9397.700000 45.581737 9327.0 9371.25 9392.5 9414.00 9508.0
    centipede 82800 20.0 9203.250000 188.459929 8697.0 9121.75 9288.0 9330.25 9438.0
    symcts 82800 19.0 1087.052632 594.948688 735.0 852.50 937.0 960.50 3133.0
    symcts_symqemu 82800 20.0 841.550000 86.139712 718.0 759.00 858.5 915.00 985.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

openssl_x509 summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    honggfuzz 82800 20.0 2319.850000 182.345616 1820.0 2234.50 2348.0 2450.75 2557.0
    centipede 82800 20.0 1894.200000 334.946594 1434.0 1717.25 1823.0 2110.00 2430.0
    symcc_aflplusplus 82800 20.0 1843.200000 114.297212 1798.0 1819.00 1823.0 1824.00 2327.0
    symcts_afl 82800 20.0 1848.900000 117.705654 1741.0 1813.50 1822.0 1823.25 2209.0
    symcts_symqemu_afl 82800 19.0 1824.631579 183.215480 1427.0 1818.00 1822.0 1823.00 2467.0
    libfuzzer 82800 20.0 2012.000000 295.787619 1817.0 1820.00 1821.0 2366.25 2547.0
    symsan 82800 7.0 1828.000000 62.077908 1765.0 1802.50 1821.0 1822.00 1961.0
    aflplusplus 82800 20.0 1909.350000 215.070191 1760.0 1815.25 1818.5 1821.50 2475.0
    symcts 82800 20.0 1787.900000 34.822709 1726.0 1771.75 1808.0 1809.00 1822.0
    symcts_symqemu 82800 19.0 1509.368421 151.611056 1400.0 1419.50 1431.0 1566.50 1804.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

openthread_ot-ip6-send-fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    symcts_afl 82800 20.0 3885.85 368.292114 3589.0 3614.25 3671.0 4253.50 4612.0
    symcts 82800 20.0 3711.05 166.591993 3563.0 3570.50 3629.0 3861.25 4036.0
    symcts_symqemu_afl 82800 20.0 3599.15 19.032590 3569.0 3589.50 3601.0 3610.25 3639.0
    symcc_aflplusplus 82800 20.0 3583.75 130.946303 3090.0 3575.00 3591.0 3609.50 3841.0
    symsan 82800 10.0 3378.90 261.453395 3064.0 3078.50 3567.5 3577.75 3606.0
    symcts_symqemu 82800 20.0 3432.85 189.890626 2987.0 3500.00 3508.5 3514.00 3542.0
    libfuzzer 82800 20.0 3095.70 18.111061 3068.0 3078.00 3102.0 3107.00 3131.0
    aflplusplus 82800 20.0 3158.25 201.326359 3038.0 3055.75 3064.5 3079.50 3609.0
    honggfuzz 82800 20.0 3132.15 253.557068 2900.0 3030.75 3056.0 3071.75 3972.0
    centipede 82800 20.0 2880.05 51.702056 2769.0 2871.50 2886.5 2909.25 2968.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

proj4_proj_crs_to_crs_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    honggfuzz 79200 20.0 7845.750000 218.122099 7440.0 7649.75 7931.5 8022.00 8125.0
    libfuzzer 79200 20.0 7404.700000 82.239125 7266.0 7346.50 7399.0 7444.50 7573.0
    symcc_aflplusplus 79200 9.0 7369.888889 102.482248 7186.0 7325.00 7366.0 7459.00 7499.0
    symcts_symqemu_afl 79200 9.0 7052.888889 147.877013 6814.0 6935.00 7093.0 7154.00 7237.0
    aflplusplus 79200 20.0 7052.350000 181.690678 6653.0 6977.25 7026.0 7165.75 7379.0
    symcts_afl 79200 2.0 6794.500000 13.435029 6785.0 6789.75 6794.5 6799.25 6804.0
    centipede 79200 20.0 820.300000 3.357631 815.0 818.50 820.0 822.00 829.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

re2_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    symcc_aflplusplus 82800 17.0 2871.588235 3.483153 2864.0 2870.00 2873.0 2873.00 2878.0
    symcts_afl 82800 3.0 2868.666667 9.712535 2858.0 2864.50 2871.0 2874.00 2877.0
    symcts_symqemu_afl 82800 20.0 2867.050000 5.472178 2855.0 2865.75 2868.0 2870.25 2878.0
    aflplusplus 82800 20.0 2866.650000 9.183939 2843.0 2864.75 2867.0 2871.00 2886.0
    honggfuzz 82800 20.0 2850.550000 7.830272 2835.0 2844.75 2851.0 2855.25 2865.0
    libfuzzer 82800 20.0 2840.200000 13.032752 2810.0 2834.00 2838.0 2845.50 2869.0
    centipede 82800 20.0 2774.250000 23.586069 2737.0 2753.75 2773.0 2792.75 2816.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

sqlite3_ossfuzz summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus 82800 20.0 10326.600000 453.479577 9593.0 10029.75 10342.0 10532.25 11431.0
    symcts_symqemu_afl 82800 3.0 9929.333333 561.612262 9281.0 9761.00 10241.0 10253.50 10266.0
    symcts_afl 82800 1.0 9626.000000 NaN 9626.0 9626.00 9626.0 9626.00 9626.0
    centipede 82800 20.0 5333.000000 1022.227907 2374.0 5087.50 5608.5 5824.75 6876.0
    libfuzzer 82800 20.0 2294.200000 1.704483 2292.0 2293.00 2294.0 2296.00 2298.0
    honggfuzz 82800 20.0 2219.700000 99.816727 2084.0 2089.50 2290.5 2291.00 2294.0
    symcts_symqemu 82800 20.0 2099.700000 73.541324 2062.0 2062.75 2064.0 2087.00 2270.0
    symcts 82800 20.0 2086.150000 85.592225 1982.0 2042.25 2054.0 2058.00 2262.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

stb_stbi_read_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    symcts_afl 82800 20.0 2026.25 121.397509 1733.0 1946.75 2018.5 2149.00 2181.0
    symcts_symqemu_afl 82800 20.0 1902.60 167.538024 1675.0 1757.00 1872.5 2088.25 2160.0
    symcc_aflplusplus 82800 4.0 1870.00 115.050713 1756.0 1779.25 1870.5 1961.25 1983.0
    aflplusplus 82800 20.0 1695.05 141.909339 1457.0 1603.25 1670.5 1757.25 1939.0
    symcts 82800 4.0 1633.00 176.699368 1476.0 1482.00 1635.0 1786.00 1786.0
    symcts_symqemu 82800 20.0 1622.20 96.078148 1403.0 1602.50 1618.0 1670.00 1822.0
    honggfuzz 82800 20.0 1395.55 80.254513 1256.0 1333.50 1388.0 1469.25 1544.0
    centipede 82800 20.0 706.40 112.881587 470.0 597.75 713.5 779.25 922.0
    libfuzzer 82800 20.0 645.30 86.699177 539.0 584.50 611.0 672.25 831.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

vorbis_decode_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    symcts_symqemu_afl 82800 17.0 1123.705882 164.413262 720.0 1112.00 1162.0 1227.00 1278.0
    symcts_afl 82800 20.0 1117.550000 149.662594 717.0 1092.00 1159.5 1204.75 1284.0
    aflplusplus 82800 20.0 1045.000000 220.373129 310.0 1082.50 1097.0 1157.50 1219.0
    libfuzzer 82800 20.0 1099.300000 104.169952 717.0 1077.00 1094.5 1143.25 1238.0
    symcts 82800 20.0 1024.700000 94.997008 713.0 1026.50 1047.5 1070.00 1138.0
    symcc_aflplusplus 82800 20.0 813.850000 318.437513 311.0 668.25 722.0 1116.75 1227.0
    symsan 82800 3.0 705.333333 390.072216 311.0 512.50 714.0 902.50 1091.0
    honggfuzz 82800 20.0 597.000000 308.545655 289.0 302.50 693.5 709.50 1127.0
    symcts_symqemu 82800 16.0 229.062500 7.487490 227.0 227.00 227.0 227.00 257.0
    centipede 82800 20.0 187.050000 43.983819 85.0 205.00 205.0 205.00 206.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

woff2_convert_woff2ttf_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libfuzzer 82800 20.0 1050.700000 13.361768 1026.0 1041.75 1048.5 1062.75 1072.0
    symcts_afl 82800 20.0 1006.800000 17.132917 974.0 1000.50 1008.0 1021.00 1032.0
    symcts_symqemu_afl 82800 20.0 1005.650000 9.847655 991.0 996.75 1007.5 1011.50 1024.0
    honggfuzz 82800 20.0 994.900000 20.897998 933.0 982.00 1000.5 1007.25 1022.0
    symcts 82800 20.0 937.150000 16.122149 904.0 927.00 935.0 949.00 964.0
    symcc_aflplusplus 82800 20.0 903.900000 135.628711 339.0 917.00 929.5 946.75 996.0
    symcts_symqemu 82800 20.0 919.800000 20.415680 876.0 908.00 917.5 937.50 950.0
    symsan 82800 7.0 801.428571 213.292332 343.0 792.50 913.0 919.50 930.0
    aflplusplus 82800 20.0 895.650000 82.469947 720.0 878.25 910.0 952.50 1006.0
    centipede 82800 20.0 888.450000 32.560026 818.0 870.50 890.5 902.25 955.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

experiment data

You can download the raw data for this report here.

Check out the documentation on how to create customized reports using this data. Also see some example Colab notebooks for doing custom analysis on the data here.

Experiment Description:

(None,)