FuzzBench: 2023-09-25-libafl-grimoire report
experiment summary
We show two different aggregate (cross-benchmark) rankings of fuzzers. The first is based on the average of per-benchmarks scores, where the score represents the percentage of the highest reached median code-coverage on a given benchmark (higher value is better). The second ranking shows the average rank of fuzzers, after we rank them on each benchmark according to their median reached code-covereges (lower value is better).-
Critical difference diagramThe diagram visualizes the average rank of fuzzers (second ranking above) while showing the significance of the differences as well. What is considered a "critical difference" (CD) is based on the Friedman/Nemenyi post-hoc test. See more in the documentation.
Note: If a fuzzer does not support all benchmarks, its ranking as shown in this diagram can be lower than it should be. So please check the list of supported benchmarks for the fuzzer(s) of your interest. The list could be specified in the fuzzer's README.md like this.
-
Median relative code-coverages on each benchmark
Note: The relative coverage summary table shows the median relative performance of each fuzzer to the experiment maximum. Thus the highest relative performance may not be 100%.
trial_relative_coverage = trial_coverage / experiment_max_coverage
libafl_fuzzbench_grimoire aflplusplus libafl libfuzzer honggfuzz eclipser mopt aflsmart afl aflfast centipede fairfuzz FuzzerMedian 96.00 98.00 95.00 92.50 95.00 93.50 92.50 93.00 92.50 92.00 86.50 84.00 FuzzerMean 93.23 92.30 89.30 87.07 85.25 84.77 78.04 77.46 76.50 75.92 75.20 73.46 arduinojson_json_fuzzer 99.00 100.00 nan nan nan nan nan nan nan nan nan nan assimp_assimp_fuzzer nan 64.00 nan 60.00 87.00 nan nan nan 50.00 nan nan nan astc-encoder_fuzz_astc_physical_to_symbolic 95.00 99.00 nan 99.00 79.00 nan nan nan 99.00 nan nan nan bloaty_fuzz_target 77.00 98.00 98.00 91.00 95.00 94.00 96.00 96.00 94.00 94.00 nan 80.00 botan_tls_server nan 48.00 nan 48.00 48.00 nan nan nan 47.00 nan nan nan brotli_decode_fuzzer 97.00 99.00 nan 98.00 nan nan nan nan 98.00 nan nan nan curl_curl_fuzzer_http 98.00 98.00 95.00 91.00 98.00 94.00 94.00 94.00 94.00 93.00 nan 83.00 double-conversion_string_to_double_fuzzer 97.00 99.00 nan nan nan nan nan nan nan nan nan nan draco_draco_pc_decoder_fuzzer nan 59.00 nan 48.00 58.00 nan nan nan 0.00 nan nan nan dropbear_fuzzer-postauth_nomaths nan 96.00 nan 95.00 98.00 nan nan nan 78.00 nan nan nan firestore_firestore_serializer_fuzzer 100.00 100.00 nan nan nan nan nan nan nan nan nan nan fmt_chrono-duration-fuzzer 98.00 99.00 nan nan nan nan nan nan nan nan nan nan freetype2_ftfuzzer 83.00 90.00 75.00 78.00 91.00 74.00 67.00 66.00 67.00 63.00 57.00 62.00 guetzli_guetzli_fuzzer 99.00 99.00 nan nan nan nan nan nan nan nan nan nan harfbuzz_hb-shape-fuzzer 81.00 98.00 99.00 94.00 95.00 97.00 96.00 97.00 96.00 95.00 nan 87.00 icu_unicode_string_codepage_create_fuzzer 97.00 99.00 nan nan nan nan nan nan nan nan nan nan jansson_json_load_dump_fuzzer 99.00 99.00 nan nan nan nan nan nan nan nan nan nan jsoncpp_jsoncpp_fuzzer 98.00 99.00 98.00 100.00 99.00 98.00 98.00 98.00 98.00 98.00 98.00 98.00 lcms_cms_transform_fuzzer 90.00 88.00 90.00 85.00 71.00 78.00 63.00 40.00 66.00 28.00 38.00 50.00 libaom_av1_dec_fuzzer 87.00 97.00 nan nan nan nan nan nan nan nan nan nan libcoap_pdu_parse_fuzzer 90.00 99.00 nan nan nan nan nan nan nan nan nan nan libhevc_hevc_dec_fuzzer 99.00 99.00 nan nan nan 0.00 0.00 0.00 nan 0.00 5.00 0.00 libjpeg-turbo_libjpeg_turbo_fuzzer 99.00 99.00 99.00 99.00 99.00 nan 82.00 82.00 99.00 82.00 82.00 82.00 libpcap_fuzz_both 86.00 92.00 76.00 79.00 nan 79.00 1.00 1.00 1.00 1.00 89.00 1.00 libpng_libpng_read_fuzzer 94.00 95.00 95.00 96.00 95.00 94.00 94.00 95.00 94.00 93.00 96.00 94.00 librdkafka_fuzz_regex nan 85.00 nan 99.00 0.00 nan nan nan 28.00 nan nan nan libxml2_xml 96.00 99.00 98.00 97.00 98.00 97.00 96.00 97.00 97.00 96.00 92.00 89.00 libxslt_xpath 96.00 98.00 96.00 92.00 97.00 95.00 93.00 95.00 94.00 93.00 93.00 96.00 mbedtls_fuzz_dtlsclient 83.00 71.00 85.00 69.00 68.00 70.00 69.00 69.00 69.00 67.00 67.00 71.00 openh264_decoder_fuzzer 97.00 99.00 99.00 98.00 99.00 99.00 99.00 99.00 99.00 99.00 nan 90.00 openssl_x509 99.00 99.00 99.00 99.00 99.00 99.00 99.00 99.00 99.00 99.00 99.00 99.00 openthread_ot-ip6-send-fuzzer 82.00 73.00 83.00 73.00 73.00 73.00 70.00 69.00 69.00 69.00 69.00 66.00 proj4_proj_crs_to_crs_fuzzer 93.00 87.00 9.00 91.00 97.00 59.00 9.00 9.00 9.00 9.00 9.00 9.00 re2_fuzzer 99.00 99.00 98.00 99.00 98.00 99.00 98.00 99.00 99.00 99.00 95.00 99.00 sqlite3_ossfuzz 82.00 89.00 91.00 80.00 71.00 93.00 92.00 93.00 91.00 90.00 64.00 65.00 stb_stbi_read_fuzzer 95.00 95.00 94.00 87.00 91.00 91.00 86.00 87.00 87.00 86.00 85.00 86.00 systemd_fuzz-link-parser 98.00 99.00 93.00 80.00 97.00 92.00 91.00 91.00 91.00 91.00 97.00 85.00 vorbis_decode_fuzzer 93.00 94.00 93.00 94.00 93.00 93.00 93.00 93.00 93.00 93.00 85.00 93.00 woff2_convert_woff2ttf_fuzzer 92.00 96.00 96.00 93.00 96.00 nan 91.00 93.00 92.00 89.00 88.00 81.00 zlib_zlib_uncompress_fuzzer 95.00 97.00 95.00 100.00 97.00 97.00 96.00 97.00 97.00 95.00 96.00 97.00 - Fuzzers are sorted by "FuzzerMean" (average median relative coverage), highest on the left.
- Green background = highest relative median coverage.
- Blue gradient background = greater than 95% relative median coverage.
arduinojson_json_fuzzer summary
Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval
around the mean code coverage.
-
Sample statistics and statistical significance (code coverage)
Code coverage sample statistics
count mean std min 25% median 75% max fuzzer time aflplusplus 82800 20.0 401.60 0.502625 401.0 401.0 402.0 402.0 402.0 libafl_fuzzbench_grimoire 82800 20.0 398.95 0.998683 397.0 398.0 399.0 400.0 401.0
Vargha-Delaney A12 measure
The table summarizes the A12 values from the
pairwise Vargha-Delaney A measure of effect size.
Green cells indicate the probability the fuzzer in the
row will outperform the fuzzer in the column.
Mann-Whitney U test
The table summarizes the p values of
pairwise Mann-Whitney U tests.
Green cells indicate that the reached
coverage distribution of a given fuzzer pair
is significantly different.
-
Unique code coverage plots
Ranking by unique code branches covered
Each bar shows the total number of code branches found by a given fuzzer.
The colored area shows the number of unique code branches
(i.e., branches that were not covered by any other fuzzers).
Pairwise unique code coverage
Each cell represents the number of code branches covered by the fuzzer
of the column but not by the fuzzer of the row
-
Code coverage reports for each fuzzer on this benchmark
assimp_assimp_fuzzer summary
Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval
around the mean code coverage.
-
Sample statistics and statistical significance (code coverage)
Code coverage sample statistics
count mean std min 25% median 75% max fuzzer time honggfuzz 3600 10.0 3133.0 280.689231 2692.0 2949.00 3100.0 3280.00 3559.0 aflplusplus 3600 10.0 2257.2 253.820934 1829.0 2128.75 2307.5 2440.75 2558.0 libfuzzer 3600 10.0 2134.0 262.037741 1484.0 2086.00 2159.5 2302.00 2423.0 afl 3600 10.0 1828.5 108.020831 1624.0 1788.50 1813.5 1899.25 1983.0
Vargha-Delaney A12 measure
The table summarizes the A12 values from the
pairwise Vargha-Delaney A measure of effect size.
Green cells indicate the probability the fuzzer in the
row will outperform the fuzzer in the column.
Mann-Whitney U test
The table summarizes the p values of
pairwise Mann-Whitney U tests.
Green cells indicate that the reached
coverage distribution of a given fuzzer pair
is significantly different.
-
Unique code coverage plots
Ranking by unique code branches covered
Each bar shows the total number of code branches found by a given fuzzer.
The colored area shows the number of unique code branches
(i.e., branches that were not covered by any other fuzzers).
Pairwise unique code coverage
Each cell represents the number of code branches covered by the fuzzer
of the column but not by the fuzzer of the row
-
Code coverage reports for each fuzzer on this benchmark
astc-encoder_fuzz_astc_physical_to_symbolic summary
Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval
around the mean code coverage.
error
The following fuzzers do not have enough samples:
afl, aflplusplus, honggfuzz, libfuzzer.
-
Sample statistics and statistical significance (code coverage)
Code coverage sample statistics
count mean std min 25% median 75% max fuzzer time afl 3600 10.0 511.6 0.699206 511.0 511.00 511.5 512.00 513.0 aflplusplus 3600 10.0 511.2 0.632456 511.0 511.00 511.0 511.00 513.0 libfuzzer 3600 10.0 511.0 0.000000 511.0 511.00 511.0 511.00 511.0 libafl_fuzzbench_grimoire 3600 20.0 489.0 0.725476 488.0 488.75 489.0 489.25 490.0 honggfuzz 3600 10.0 414.5 14.416426 406.0 406.75 409.0 409.00 448.0
Vargha-Delaney A12 measure
The table summarizes the A12 values from the
pairwise Vargha-Delaney A measure of effect size.
Green cells indicate the probability the fuzzer in the
row will outperform the fuzzer in the column.
Mann-Whitney U test
The table summarizes the p values of
pairwise Mann-Whitney U tests.
Green cells indicate that the reached
coverage distribution of a given fuzzer pair
is significantly different.
-
Unique code coverage plots
Ranking by unique code branches covered
Each bar shows the total number of code branches found by a given fuzzer.
The colored area shows the number of unique code branches
(i.e., branches that were not covered by any other fuzzers).
Pairwise unique code coverage
Each cell represents the number of code branches covered by the fuzzer
of the column but not by the fuzzer of the row
-
Code coverage reports for each fuzzer on this benchmark
bloaty_fuzz_target summary
Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval
around the mean code coverage.
-
Sample statistics and statistical significance (code coverage)
Code coverage sample statistics
count mean std min 25% median 75% max fuzzer time libafl 82800 20.0 6353.80 71.997515 6140.0 6344.75 6371.5 6388.00 6442.0 aflplusplus 82800 20.0 6356.00 57.764495 6145.0 6345.50 6358.5 6393.00 6415.0 mopt 82800 20.0 6236.05 71.516266 6071.0 6226.50 6243.5 6268.50 6354.0 aflsmart 82800 20.0 6198.45 127.984981 5937.0 6102.25 6205.0 6295.25 6394.0 honggfuzz 82800 20.0 6100.50 209.354522 5744.0 5948.25 6131.5 6295.50 6391.0 eclipser 82800 20.0 6110.20 114.487232 5900.0 6066.75 6107.0 6144.50 6360.0 afl 82800 20.0 6108.30 77.916352 5889.0 6081.25 6106.0 6148.00 6291.0 aflfast 82800 20.0 6080.10 119.230824 5858.0 6033.50 6059.0 6136.50 6350.0 libfuzzer 82800 20.0 5897.05 169.163414 5559.0 5805.00 5890.5 5988.00 6182.0 fairfuzz 82800 20.0 5217.75 121.813868 5060.0 5145.00 5194.5 5272.75 5570.0 libafl_fuzzbench_grimoire 82800 20.0 4972.55 47.526696 4910.0 4944.25 4967.0 4992.50 5097.0
Vargha-Delaney A12 measure
The table summarizes the A12 values from the
pairwise Vargha-Delaney A measure of effect size.
Green cells indicate the probability the fuzzer in the
row will outperform the fuzzer in the column.
Mann-Whitney U test
The table summarizes the p values of
pairwise Mann-Whitney U tests.
Green cells indicate that the reached
coverage distribution of a given fuzzer pair
is significantly different.
-
Unique code coverage plots
Ranking by unique code branches covered
Each bar shows the total number of code branches found by a given fuzzer.
The colored area shows the number of unique code branches
(i.e., branches that were not covered by any other fuzzers).
Pairwise unique code coverage
Each cell represents the number of code branches covered by the fuzzer
of the column but not by the fuzzer of the row
-
Code coverage reports for each fuzzer on this benchmark
botan_tls_server summary
Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval
around the mean code coverage.
error
The following fuzzers do not have enough samples:
afl, honggfuzz, libfuzzer.
-
Sample statistics and statistical significance (code coverage)
Code coverage sample statistics
count mean std min 25% median 75% max fuzzer time honggfuzz 3600 10.0 1365.1 188.481328 1171.0 1242.75 1252.5 1549.00 1671.0 aflplusplus 3600 20.0 1333.2 203.259906 1179.0 1224.00 1252.0 1288.75 1792.0 libfuzzer 3600 10.0 1217.0 34.062035 1176.0 1178.25 1241.5 1242.00 1248.0 afl 3600 10.0 1239.9 6.172520 1235.0 1236.25 1237.5 1241.25 1255.0
Vargha-Delaney A12 measure
The table summarizes the A12 values from the
pairwise Vargha-Delaney A measure of effect size.
Green cells indicate the probability the fuzzer in the
row will outperform the fuzzer in the column.
Mann-Whitney U test
The table summarizes the p values of
pairwise Mann-Whitney U tests.
Green cells indicate that the reached
coverage distribution of a given fuzzer pair
is significantly different.
-
Unique code coverage plots
Ranking by unique code branches covered
Each bar shows the total number of code branches found by a given fuzzer.
The colored area shows the number of unique code branches
(i.e., branches that were not covered by any other fuzzers).
Pairwise unique code coverage
Each cell represents the number of code branches covered by the fuzzer
of the column but not by the fuzzer of the row
-
Code coverage reports for each fuzzer on this benchmark
brotli_decode_fuzzer summary
Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval
around the mean code coverage.
error
The following fuzzers do not have enough samples:
afl, aflplusplus, libfuzzer.
-
Sample statistics and statistical significance (code coverage)
Code coverage sample statistics
count mean std min 25% median 75% max fuzzer time aflplusplus 3600 10.0 898.2 3.326660 894.0 894.50 900.0 900.00 902.0 afl 3600 10.0 888.5 6.485025 876.0 883.75 890.5 892.50 897.0 libfuzzer 3600 10.0 889.3 6.342975 878.0 889.25 890.5 891.00 897.0 libafl_fuzzbench_grimoire 3600 20.0 880.7 4.878524 875.0 877.75 879.0 886.25 889.0
Vargha-Delaney A12 measure
The table summarizes the A12 values from the
pairwise Vargha-Delaney A measure of effect size.
Green cells indicate the probability the fuzzer in the
row will outperform the fuzzer in the column.
Mann-Whitney U test
The table summarizes the p values of
pairwise Mann-Whitney U tests.
Green cells indicate that the reached
coverage distribution of a given fuzzer pair
is significantly different.
-
Unique code coverage plots
Ranking by unique code branches covered
Each bar shows the total number of code branches found by a given fuzzer.
The colored area shows the number of unique code branches
(i.e., branches that were not covered by any other fuzzers).
Pairwise unique code coverage
Each cell represents the number of code branches covered by the fuzzer
of the column but not by the fuzzer of the row
-
Code coverage reports for each fuzzer on this benchmark
curl_curl_fuzzer_http summary
Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval
around the mean code coverage.
-
Sample statistics and statistical significance (code coverage)
Code coverage sample statistics
count mean std min 25% median 75% max fuzzer time aflplusplus 82800 20.0 10864.95 64.618292 10747.0 10829.75 10877.0 10894.75 11006.0 honggfuzz 82800 20.0 10845.70 108.334327 10689.0 10747.75 10861.0 10926.00 11036.0 libafl_fuzzbench_grimoire 82800 20.0 10846.95 33.512723 10799.0 10821.25 10843.0 10872.50 10912.0 libafl 82800 20.0 10509.65 42.703722 10446.0 10485.50 10501.0 10527.00 10604.0 eclipser 82800 20.0 10463.90 37.160959 10415.0 10437.00 10454.0 10491.00 10551.0 aflsmart 82800 20.0 10456.75 63.366831 10389.0 10399.00 10436.5 10486.75 10575.0 afl 82800 20.0 10358.75 115.808769 10055.0 10329.50 10384.5 10432.75 10482.0 mopt 82800 20.0 10338.90 115.764234 10036.0 10331.75 10376.0 10408.50 10441.0 aflfast 82800 20.0 10272.85 106.610592 9985.0 10221.25 10305.5 10346.25 10406.0 libfuzzer 82800 20.0 9985.20 396.894206 9224.0 9836.25 10063.0 10172.00 10585.0 fairfuzz 82800 20.0 9200.80 420.958005 7990.0 9035.75 9266.0 9472.50 9906.0
Vargha-Delaney A12 measure
The table summarizes the A12 values from the
pairwise Vargha-Delaney A measure of effect size.
Green cells indicate the probability the fuzzer in the
row will outperform the fuzzer in the column.
Mann-Whitney U test
The table summarizes the p values of
pairwise Mann-Whitney U tests.
Green cells indicate that the reached
coverage distribution of a given fuzzer pair
is significantly different.
-
Unique code coverage plots
Ranking by unique code branches covered
Each bar shows the total number of code branches found by a given fuzzer.
The colored area shows the number of unique code branches
(i.e., branches that were not covered by any other fuzzers).
Pairwise unique code coverage
Each cell represents the number of code branches covered by the fuzzer
of the column but not by the fuzzer of the row
-
Code coverage reports for each fuzzer on this benchmark
double-conversion_string_to_double_fuzzer summary
Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval
around the mean code coverage.
-
Sample statistics and statistical significance (code coverage)
Code coverage sample statistics
count mean std min 25% median 75% max fuzzer time aflplusplus 82800 20.0 507.75 1.831738 504.0 506.0 508.0 509.00 510.0 libafl_fuzzbench_grimoire 82800 20.0 498.75 2.899637 492.0 497.0 499.0 500.25 505.0
Vargha-Delaney A12 measure
The table summarizes the A12 values from the
pairwise Vargha-Delaney A measure of effect size.
Green cells indicate the probability the fuzzer in the
row will outperform the fuzzer in the column.
Mann-Whitney U test
The table summarizes the p values of
pairwise Mann-Whitney U tests.
Green cells indicate that the reached
coverage distribution of a given fuzzer pair
is significantly different.
-
Unique code coverage plots
Ranking by unique code branches covered
Each bar shows the total number of code branches found by a given fuzzer.
The colored area shows the number of unique code branches
(i.e., branches that were not covered by any other fuzzers).
Pairwise unique code coverage
Each cell represents the number of code branches covered by the fuzzer
of the column but not by the fuzzer of the row
-
Code coverage reports for each fuzzer on this benchmark
draco_draco_pc_decoder_fuzzer summary
Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval
around the mean code coverage.
error
The following fuzzers do not have enough samples:
afl, honggfuzz, libfuzzer.
-
Sample statistics and statistical significance (code coverage)
Code coverage sample statistics
count mean std min 25% median 75% max fuzzer time aflplusplus 3600 20.0 1187.7 174.849740 935.0 1036.00 1193.0 1339.50 1495.0 honggfuzz 3600 10.0 1165.0 78.646890 1005.0 1119.75 1175.5 1227.00 1263.0 libfuzzer 3600 10.0 950.3 93.641218 805.0 893.50 967.5 1026.25 1051.0 afl 3600 10.0 7.0 0.000000 7.0 7.00 7.0 7.00 7.0
Vargha-Delaney A12 measure
The table summarizes the A12 values from the
pairwise Vargha-Delaney A measure of effect size.
Green cells indicate the probability the fuzzer in the
row will outperform the fuzzer in the column.
Mann-Whitney U test
The table summarizes the p values of
pairwise Mann-Whitney U tests.
Green cells indicate that the reached
coverage distribution of a given fuzzer pair
is significantly different.
-
Unique code coverage plots
Ranking by unique code branches covered
Each bar shows the total number of code branches found by a given fuzzer.
The colored area shows the number of unique code branches
(i.e., branches that were not covered by any other fuzzers).
Pairwise unique code coverage
Each cell represents the number of code branches covered by the fuzzer
of the column but not by the fuzzer of the row
-
Code coverage reports for each fuzzer on this benchmark
dropbear_fuzzer-postauth_nomaths summary
Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval
around the mean code coverage.
-
Sample statistics and statistical significance (code coverage)
Code coverage sample statistics
count mean std min 25% median 75% max fuzzer time honggfuzz 3600 10.0 883.6 9.570789 860.0 881.25 885.5 889.0 895.0 aflplusplus 3600 10.0 863.0 2.538591 858.0 861.00 864.5 865.0 865.0 libfuzzer 3600 10.0 857.1 5.877452 841.0 857.00 858.5 860.0 861.0 afl 3600 10.0 701.0 0.000000 701.0 701.00 701.0 701.0 701.0
Vargha-Delaney A12 measure
The table summarizes the A12 values from the
pairwise Vargha-Delaney A measure of effect size.
Green cells indicate the probability the fuzzer in the
row will outperform the fuzzer in the column.
Mann-Whitney U test
The table summarizes the p values of
pairwise Mann-Whitney U tests.
Green cells indicate that the reached
coverage distribution of a given fuzzer pair
is significantly different.
-
Unique code coverage plots
Ranking by unique code branches covered
Each bar shows the total number of code branches found by a given fuzzer.
The colored area shows the number of unique code branches
(i.e., branches that were not covered by any other fuzzers).
Pairwise unique code coverage
Each cell represents the number of code branches covered by the fuzzer
of the column but not by the fuzzer of the row
-
Code coverage reports for each fuzzer on this benchmark
firestore_firestore_serializer_fuzzer summary
Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval
around the mean code coverage.
-
Sample statistics and statistical significance (code coverage)
Code coverage sample statistics
count mean std min 25% median 75% max fuzzer time aflplusplus 82800 20.0 287.0 0.000000 287.0 287.0 287.0 287.0 287.0 libafl_fuzzbench_grimoire 82800 20.0 286.4 0.940325 285.0 285.0 287.0 287.0 287.0
Vargha-Delaney A12 measure
The table summarizes the A12 values from the
pairwise Vargha-Delaney A measure of effect size.
Green cells indicate the probability the fuzzer in the
row will outperform the fuzzer in the column.
Mann-Whitney U test
The table summarizes the p values of
pairwise Mann-Whitney U tests.
Green cells indicate that the reached
coverage distribution of a given fuzzer pair
is significantly different.
-
Unique code coverage plots
Ranking by unique code branches covered
Each bar shows the total number of code branches found by a given fuzzer.
The colored area shows the number of unique code branches
(i.e., branches that were not covered by any other fuzzers).
Pairwise unique code coverage
Each cell represents the number of code branches covered by the fuzzer
of the column but not by the fuzzer of the row
-
Code coverage reports for each fuzzer on this benchmark
fmt_chrono-duration-fuzzer summary
Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval
around the mean code coverage.
-
Sample statistics and statistical significance (code coverage)
Code coverage sample statistics
count mean std min 25% median 75% max fuzzer time aflplusplus 82800 20.0 1089.80 3.607011 1081.0 1086.75 1091.0 1093.00 1095.0 libafl_fuzzbench_grimoire 82800 20.0 1080.35 3.660242 1073.0 1078.00 1080.0 1083.25 1086.0
Vargha-Delaney A12 measure
The table summarizes the A12 values from the
pairwise Vargha-Delaney A measure of effect size.
Green cells indicate the probability the fuzzer in the
row will outperform the fuzzer in the column.
Mann-Whitney U test
The table summarizes the p values of
pairwise Mann-Whitney U tests.
Green cells indicate that the reached
coverage distribution of a given fuzzer pair
is significantly different.
-
Unique code coverage plots
Ranking by unique code branches covered
Each bar shows the total number of code branches found by a given fuzzer.
The colored area shows the number of unique code branches
(i.e., branches that were not covered by any other fuzzers).
Pairwise unique code coverage
Each cell represents the number of code branches covered by the fuzzer
of the column but not by the fuzzer of the row
-
Code coverage reports for each fuzzer on this benchmark
freetype2_ftfuzzer summary
Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval
around the mean code coverage.
-
Sample statistics and statistical significance (code coverage)
Code coverage sample statistics
count mean std min 25% median 75% max fuzzer time honggfuzz 82800 20.0 11253.85 623.984925 9786.0 10822.25 11416.5 11720.50 12085.0 aflplusplus 82800 20.0 11391.20 567.364229 10599.0 10947.00 11322.0 11623.50 12483.0 libafl_fuzzbench_grimoire 82800 20.0 10384.10 525.427235 9267.0 10107.00 10371.5 10525.25 11721.0 libfuzzer 82800 20.0 9845.70 650.336035 8730.0 9404.00 9806.0 10147.75 11277.0 libafl 82800 20.0 9370.55 445.269224 8602.0 9007.75 9460.0 9741.75 10175.0 eclipser 82800 20.0 9242.10 100.454703 9028.0 9198.75 9263.0 9316.75 9423.0 mopt 82800 20.0 8332.40 177.225815 7888.0 8320.75 8377.5 8442.00 8504.0 afl 82800 20.0 8317.15 168.047385 7931.0 8288.25 8365.0 8420.75 8496.0 aflsmart 82800 20.0 8244.00 202.383690 7800.0 8230.50 8304.5 8374.50 8524.0 aflfast 82800 20.0 7976.35 222.903159 7640.0 7820.50 7916.0 8204.75 8311.0 fairfuzz 82800 20.0 7832.95 112.700570 7683.0 7768.00 7816.0 7859.75 8213.0 centipede 82800 20.0 7160.40 177.924112 6873.0 7047.00 7140.5 7259.00 7535.0
Vargha-Delaney A12 measure
The table summarizes the A12 values from the
pairwise Vargha-Delaney A measure of effect size.
Green cells indicate the probability the fuzzer in the
row will outperform the fuzzer in the column.
Mann-Whitney U test
The table summarizes the p values of
pairwise Mann-Whitney U tests.
Green cells indicate that the reached
coverage distribution of a given fuzzer pair
is significantly different.
-
Unique code coverage plots
Ranking by unique code branches covered
Each bar shows the total number of code branches found by a given fuzzer.
The colored area shows the number of unique code branches
(i.e., branches that were not covered by any other fuzzers).
Pairwise unique code coverage
Each cell represents the number of code branches covered by the fuzzer
of the column but not by the fuzzer of the row
-
Code coverage reports for each fuzzer on this benchmark
guetzli_guetzli_fuzzer summary
Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval
around the mean code coverage.
-
Sample statistics and statistical significance (code coverage)
Code coverage sample statistics
count mean std min 25% median 75% max fuzzer time aflplusplus 82800 20.0 1498.30 5.391026 1484.0 1495.00 1498.5 1502.25 1506.0 libafl_fuzzbench_grimoire 82800 20.0 1491.85 2.109502 1489.0 1490.75 1491.5 1493.00 1497.0
Vargha-Delaney A12 measure
The table summarizes the A12 values from the
pairwise Vargha-Delaney A measure of effect size.
Green cells indicate the probability the fuzzer in the
row will outperform the fuzzer in the column.
Mann-Whitney U test
The table summarizes the p values of
pairwise Mann-Whitney U tests.
Green cells indicate that the reached
coverage distribution of a given fuzzer pair
is significantly different.
-
Unique code coverage plots
Ranking by unique code branches covered
Each bar shows the total number of code branches found by a given fuzzer.
The colored area shows the number of unique code branches
(i.e., branches that were not covered by any other fuzzers).
Pairwise unique code coverage
Each cell represents the number of code branches covered by the fuzzer
of the column but not by the fuzzer of the row
-
Code coverage reports for each fuzzer on this benchmark
harfbuzz_hb-shape-fuzzer summary
Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval
around the mean code coverage.
error
The following fuzzers do not have enough samples:
honggfuzz.
-
Sample statistics and statistical significance (code coverage)
Code coverage sample statistics
count mean std min 25% median 75% max fuzzer time libafl 82800 20.0 11091.70 21.812840 11042.0 11078.75 11088.5 11103.00 11139.0 aflplusplus 82800 20.0 10907.05 305.379570 9630.0 10940.25 10973.0 11022.25 11067.0 aflsmart 82800 20.0 10815.45 39.488139 10703.0 10800.25 10821.0 10841.25 10879.0 eclipser 82800 20.0 10820.15 43.008292 10754.0 10778.50 10819.5 10861.50 10885.0 mopt 82800 20.0 10796.70 40.874777 10699.0 10777.25 10801.5 10821.00 10864.0 afl 82800 20.0 10749.75 54.151712 10644.0 10700.00 10766.0 10787.50 10833.0 honggfuzz 82800 5.0 10670.60 74.214554 10560.0 10642.00 10682.0 10716.00 10753.0 aflfast 82800 20.0 10670.70 51.265152 10567.0 10642.75 10669.0 10702.75 10770.0 libfuzzer 82800 20.0 10539.65 52.400557 10425.0 10503.50 10541.5 10578.25 10630.0 fairfuzz 82800 20.0 9728.25 323.424257 9130.0 9481.50 9724.0 9986.00 10218.0 libafl_fuzzbench_grimoire 82800 20.0 9047.55 95.023252 8900.0 8966.00 9053.0 9104.50 9254.0
Vargha-Delaney A12 measure
The table summarizes the A12 values from the
pairwise Vargha-Delaney A measure of effect size.
Green cells indicate the probability the fuzzer in the
row will outperform the fuzzer in the column.
Mann-Whitney U test
The table summarizes the p values of
pairwise Mann-Whitney U tests.
Green cells indicate that the reached
coverage distribution of a given fuzzer pair
is significantly different.
-
Unique code coverage plots
Ranking by unique code branches covered
Each bar shows the total number of code branches found by a given fuzzer.
The colored area shows the number of unique code branches
(i.e., branches that were not covered by any other fuzzers).
Pairwise unique code coverage
Each cell represents the number of code branches covered by the fuzzer
of the column but not by the fuzzer of the row
-
Code coverage reports for each fuzzer on this benchmark
icu_unicode_string_codepage_create_fuzzer summary
Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval
around the mean code coverage.
-
Sample statistics and statistical significance (code coverage)
Code coverage sample statistics
count mean std min 25% median 75% max fuzzer time aflplusplus 82800 20.0 1339.85 0.74516 1339.0 1339.0 1340.0 1340.00 1341.0 libafl_fuzzbench_grimoire 82800 20.0 1312.20 17.13906 1298.0 1300.0 1301.0 1328.75 1340.0
Vargha-Delaney A12 measure
The table summarizes the A12 values from the
pairwise Vargha-Delaney A measure of effect size.
Green cells indicate the probability the fuzzer in the
row will outperform the fuzzer in the column.
Mann-Whitney U test
The table summarizes the p values of
pairwise Mann-Whitney U tests.
Green cells indicate that the reached
coverage distribution of a given fuzzer pair
is significantly different.
-
Unique code coverage plots
Ranking by unique code branches covered
Each bar shows the total number of code branches found by a given fuzzer.
The colored area shows the number of unique code branches
(i.e., branches that were not covered by any other fuzzers).
Pairwise unique code coverage
Each cell represents the number of code branches covered by the fuzzer
of the column but not by the fuzzer of the row
-
Code coverage reports for each fuzzer on this benchmark
jansson_json_load_dump_fuzzer summary
Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval
around the mean code coverage.
-
Sample statistics and statistical significance (code coverage)
Code coverage sample statistics
count mean std min 25% median 75% max fuzzer time aflplusplus 82800 20.0 767.8 1.472556 765.0 767.0 768.0 769.0 770.0 libafl_fuzzbench_grimoire 82800 20.0 766.9 1.713722 764.0 766.0 767.0 768.0 770.0
Vargha-Delaney A12 measure
The table summarizes the A12 values from the
pairwise Vargha-Delaney A measure of effect size.
Green cells indicate the probability the fuzzer in the
row will outperform the fuzzer in the column.
Mann-Whitney U test
The table summarizes the p values of
pairwise Mann-Whitney U tests.
Green cells indicate that the reached
coverage distribution of a given fuzzer pair
is significantly different.
-
Unique code coverage plots
Ranking by unique code branches covered
Each bar shows the total number of code branches found by a given fuzzer.
The colored area shows the number of unique code branches
(i.e., branches that were not covered by any other fuzzers).
Pairwise unique code coverage
Each cell represents the number of code branches covered by the fuzzer
of the column but not by the fuzzer of the row
-
Code coverage reports for each fuzzer on this benchmark
jsoncpp_jsoncpp_fuzzer summary
Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval
around the mean code coverage.
-
Sample statistics and statistical significance (code coverage)
Code coverage sample statistics
count mean std min 25% median 75% max fuzzer time libfuzzer 82800 20.0 524.95 0.223607 524.0 525.00 525.0 525.00 525.0 honggfuzz 82800 20.0 522.45 1.356272 519.0 522.00 522.0 524.00 524.0 aflplusplus 82800 20.0 520.00 0.000000 520.0 520.00 520.0 520.00 520.0 eclipser 82800 20.0 518.90 1.252366 516.0 518.00 519.5 520.00 520.0 centipede 82800 20.0 519.65 1.980829 517.0 518.75 519.0 520.00 524.0 mopt 82800 20.0 517.20 3.955010 505.0 518.00 518.0 519.00 520.0 aflsmart 82800 20.0 517.90 1.165287 516.0 517.00 517.5 519.00 520.0 afl 82800 20.0 517.65 0.812728 517.0 517.00 517.0 518.00 519.0 aflfast 82800 20.0 513.90 10.622221 478.0 517.00 517.0 519.00 519.0 fairfuzz 82800 20.0 517.45 0.998683 516.0 517.00 517.0 517.25 520.0 libafl 82800 20.0 517.10 0.911910 516.0 517.00 517.0 517.00 519.0 libafl_fuzzbench_grimoire 82800 20.0 517.30 0.732695 516.0 517.00 517.0 518.00 518.0
Vargha-Delaney A12 measure
The table summarizes the A12 values from the
pairwise Vargha-Delaney A measure of effect size.
Green cells indicate the probability the fuzzer in the
row will outperform the fuzzer in the column.
Mann-Whitney U test
The table summarizes the p values of
pairwise Mann-Whitney U tests.
Green cells indicate that the reached
coverage distribution of a given fuzzer pair
is significantly different.
-
Unique code coverage plots
Ranking by unique code branches covered
Each bar shows the total number of code branches found by a given fuzzer.
The colored area shows the number of unique code branches
(i.e., branches that were not covered by any other fuzzers).
Pairwise unique code coverage
Each cell represents the number of code branches covered by the fuzzer
of the column but not by the fuzzer of the row
-
Code coverage reports for each fuzzer on this benchmark
lcms_cms_transform_fuzzer summary
Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval
around the mean code coverage.
-
Sample statistics and statistical significance (code coverage)
Code coverage sample statistics
count mean std min 25% median 75% max fuzzer time libafl_fuzzbench_grimoire 82800 20.0 2052.50 67.065169 1932.0 2011.50 2059.5 2079.75 2186.0 libafl 82800 20.0 2044.70 142.437466 1553.0 2021.00 2057.5 2104.25 2284.0 aflplusplus 82800 20.0 1935.25 210.561124 1540.0 1898.75 2025.5 2071.50 2128.0 libfuzzer 82800 20.0 1946.00 97.029024 1790.0 1889.75 1947.5 1983.25 2140.0 eclipser 82800 20.0 1736.00 135.909025 1522.0 1573.00 1789.0 1838.50 1904.0 honggfuzz 82800 20.0 1377.20 536.015279 686.0 726.25 1623.5 1811.00 2030.0 afl 82800 20.0 1260.10 448.102890 648.0 795.50 1510.5 1612.75 1812.0 mopt 82800 20.0 1230.00 430.050425 651.0 845.25 1439.5 1604.00 1765.0 fairfuzz 82800 20.0 1181.85 428.681488 652.0 790.25 1148.5 1575.75 1919.0 aflsmart 82800 20.0 1123.25 441.957235 649.0 673.75 918.0 1575.50 1795.0 centipede 82800 20.0 1058.50 289.194617 762.0 797.75 890.0 1360.50 1542.0 aflfast 82800 20.0 635.60 19.277475 569.0 637.75 642.5 646.00 657.0
Vargha-Delaney A12 measure
The table summarizes the A12 values from the
pairwise Vargha-Delaney A measure of effect size.
Green cells indicate the probability the fuzzer in the
row will outperform the fuzzer in the column.
Mann-Whitney U test
The table summarizes the p values of
pairwise Mann-Whitney U tests.
Green cells indicate that the reached
coverage distribution of a given fuzzer pair
is significantly different.
-
Unique code coverage plots
Ranking by unique code branches covered
Each bar shows the total number of code branches found by a given fuzzer.
The colored area shows the number of unique code branches
(i.e., branches that were not covered by any other fuzzers).
Pairwise unique code coverage
Each cell represents the number of code branches covered by the fuzzer
of the column but not by the fuzzer of the row
-
Code coverage reports for each fuzzer on this benchmark
libaom_av1_dec_fuzzer summary
Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval
around the mean code coverage.
-
Sample statistics and statistical significance (code coverage)
Code coverage sample statistics
count mean std min 25% median 75% max fuzzer time aflplusplus 82800 20.0 11048.45 295.341457 10282.0 10997.75 11104.5 11242.25 11352.0 libafl_fuzzbench_grimoire 82800 20.0 9965.20 65.273997 9865.0 9922.00 9945.5 10005.25 10112.0
Vargha-Delaney A12 measure
The table summarizes the A12 values from the
pairwise Vargha-Delaney A measure of effect size.
Green cells indicate the probability the fuzzer in the
row will outperform the fuzzer in the column.
Mann-Whitney U test
The table summarizes the p values of
pairwise Mann-Whitney U tests.
Green cells indicate that the reached
coverage distribution of a given fuzzer pair
is significantly different.
-
Unique code coverage plots
Ranking by unique code branches covered
Each bar shows the total number of code branches found by a given fuzzer.
The colored area shows the number of unique code branches
(i.e., branches that were not covered by any other fuzzers).
Pairwise unique code coverage
Each cell represents the number of code branches covered by the fuzzer
of the column but not by the fuzzer of the row
-
Code coverage reports for each fuzzer on this benchmark
libcoap_pdu_parse_fuzzer summary
Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval
around the mean code coverage.
-
Sample statistics and statistical significance (code coverage)
Code coverage sample statistics
count mean std min 25% median 75% max fuzzer time aflplusplus 82800 20.0 817.35 1.268028 815.0 817.0 817.0 818.00 821.0 libafl_fuzzbench_grimoire 82800 20.0 750.30 15.016132 746.0 746.0 747.0 747.25 814.0
Vargha-Delaney A12 measure
The table summarizes the A12 values from the
pairwise Vargha-Delaney A measure of effect size.
Green cells indicate the probability the fuzzer in the
row will outperform the fuzzer in the column.
Mann-Whitney U test
The table summarizes the p values of
pairwise Mann-Whitney U tests.
Green cells indicate that the reached
coverage distribution of a given fuzzer pair
is significantly different.
-
Unique code coverage plots
Ranking by unique code branches covered
Each bar shows the total number of code branches found by a given fuzzer.
The colored area shows the number of unique code branches
(i.e., branches that were not covered by any other fuzzers).
Pairwise unique code coverage
Each cell represents the number of code branches covered by the fuzzer
of the column but not by the fuzzer of the row
-
Code coverage reports for each fuzzer on this benchmark
libhevc_hevc_dec_fuzzer summary
Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval
around the mean code coverage.
-
Sample statistics and statistical significance (code coverage)
Code coverage sample statistics
count mean std min 25% median 75% max fuzzer time aflplusplus 82800 20.0 10332.7 47.309841 10219.0 10331.50 10355.0 10360.00 10375.0 libafl_fuzzbench_grimoire 82800 20.0 10301.5 15.274678 10285.0 10287.75 10298.0 10309.75 10332.0 centipede 82800 20.0 1260.3 1723.585892 555.0 555.00 557.0 557.00 5651.0 aflfast 82800 20.0 0.0 0.000000 0.0 0.00 0.0 0.00 0.0 aflsmart 82800 20.0 0.0 0.000000 0.0 0.00 0.0 0.00 0.0 eclipser 82800 17.0 0.0 0.000000 0.0 0.00 0.0 0.00 0.0 fairfuzz 82800 20.0 0.0 0.000000 0.0 0.00 0.0 0.00 0.0 mopt 82800 20.0 0.0 0.000000 0.0 0.00 0.0 0.00 0.0
Vargha-Delaney A12 measure
The table summarizes the A12 values from the
pairwise Vargha-Delaney A measure of effect size.
Green cells indicate the probability the fuzzer in the
row will outperform the fuzzer in the column.
Mann-Whitney U test
The table summarizes the p values of
pairwise Mann-Whitney U tests.
Green cells indicate that the reached
coverage distribution of a given fuzzer pair
is significantly different.
-
Unique code coverage plots
Ranking by unique code branches covered
Each bar shows the total number of code branches found by a given fuzzer.
The colored area shows the number of unique code branches
(i.e., branches that were not covered by any other fuzzers).
Pairwise unique code coverage
Each cell represents the number of code branches covered by the fuzzer
of the column but not by the fuzzer of the row
-
Code coverage reports for each fuzzer on this benchmark
libjpeg-turbo_libjpeg_turbo_fuzzer summary
Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval
around the mean code coverage.
-
Sample statistics and statistical significance (code coverage)
Code coverage sample statistics
count mean std min 25% median 75% max fuzzer time libfuzzer 82800 20.0 3086.20 1.507874 3083.0 3085.00 3087.0 3087.00 3088.0 aflplusplus 82800 20.0 3078.20 15.763382 3013.0 3080.00 3081.5 3084.00 3087.0 libafl 82800 20.0 3079.75 8.872637 3044.0 3079.50 3081.5 3083.25 3086.0 afl 82800 20.0 3071.50 20.464346 3007.0 3071.75 3080.0 3083.00 3086.0 libafl_fuzzbench_grimoire 82800 20.0 3055.85 19.794404 3028.0 3035.25 3065.5 3072.50 3081.0 honggfuzz 82800 20.0 3057.05 19.465151 3007.0 3055.25 3064.0 3067.00 3074.0 centipede 82800 20.0 2546.05 0.686333 2545.0 2546.00 2546.0 2546.25 2547.0 aflsmart 82800 20.0 2545.35 1.980829 2543.0 2544.00 2545.0 2546.00 2551.0 mopt 82800 20.0 2544.45 1.099043 2543.0 2544.00 2544.0 2545.00 2547.0 aflfast 82800 20.0 2543.90 1.483240 2541.0 2543.00 2543.5 2545.00 2548.0 fairfuzz 82800 20.0 2539.95 4.795557 2533.0 2536.00 2542.0 2542.25 2550.0
Vargha-Delaney A12 measure
The table summarizes the A12 values from the
pairwise Vargha-Delaney A measure of effect size.
Green cells indicate the probability the fuzzer in the
row will outperform the fuzzer in the column.
Mann-Whitney U test
The table summarizes the p values of
pairwise Mann-Whitney U tests.
Green cells indicate that the reached
coverage distribution of a given fuzzer pair
is significantly different.
-
Unique code coverage plots
Ranking by unique code branches covered
Each bar shows the total number of code branches found by a given fuzzer.
The colored area shows the number of unique code branches
(i.e., branches that were not covered by any other fuzzers).
Pairwise unique code coverage
Each cell represents the number of code branches covered by the fuzzer
of the column but not by the fuzzer of the row
-
Code coverage reports for each fuzzer on this benchmark
libpcap_fuzz_both summary
Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval
around the mean code coverage.
-
Sample statistics and statistical significance (code coverage)
Code coverage sample statistics
count mean std min 25% median 75% max fuzzer time aflplusplus 82800 20.0 2964.550000 109.551466 2790.0 2865.25 2970.5 3029.25 3218.0 centipede 82800 20.0 2693.100000 641.869219 101.0 2738.50 2876.0 2976.00 3138.0 libafl_fuzzbench_grimoire 82800 19.0 2766.578947 133.749898 2568.0 2682.50 2775.0 2833.00 3081.0 libfuzzer 82800 20.0 2521.050000 151.141090 2104.0 2494.25 2562.5 2597.75 2712.0 eclipser 82800 20.0 2506.050000 205.275547 1990.0 2411.75 2553.5 2590.75 2872.0 libafl 82800 20.0 2397.800000 221.079790 1991.0 2324.50 2448.0 2513.25 2721.0 aflfast 82800 20.0 40.300000 4.231461 34.0 34.00 43.0 43.00 43.0 fairfuzz 82800 20.0 38.850000 4.404244 33.0 33.00 42.0 42.00 42.0 afl 82800 20.0 37.650000 4.246051 34.0 34.00 34.0 43.00 43.0 mopt 82800 20.0 37.400000 4.357691 34.0 34.00 34.0 43.00 43.0 aflsmart 82800 20.0 34.800000 3.693522 33.0 33.00 33.0 33.00 42.0
Vargha-Delaney A12 measure
The table summarizes the A12 values from the
pairwise Vargha-Delaney A measure of effect size.
Green cells indicate the probability the fuzzer in the
row will outperform the fuzzer in the column.
Mann-Whitney U test
The table summarizes the p values of
pairwise Mann-Whitney U tests.
Green cells indicate that the reached
coverage distribution of a given fuzzer pair
is significantly different.
-
Unique code coverage plots
Ranking by unique code branches covered
Each bar shows the total number of code branches found by a given fuzzer.
The colored area shows the number of unique code branches
(i.e., branches that were not covered by any other fuzzers).
Pairwise unique code coverage
Each cell represents the number of code branches covered by the fuzzer
of the column but not by the fuzzer of the row
-
Code coverage reports for each fuzzer on this benchmark
libpng_libpng_read_fuzzer summary
Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval
around the mean code coverage.
-
Sample statistics and statistical significance (code coverage)
Code coverage sample statistics
count mean std min 25% median 75% max fuzzer time libfuzzer 82800 20.0 2018.00 1.169795 2015.0 2018.00 2018.0 2019.00 2020.0 centipede 82800 20.0 2015.20 1.704483 2012.0 2013.75 2016.0 2017.00 2017.0 honggfuzz 82800 20.0 2028.05 33.887546 2003.0 2010.00 2012.5 2017.25 2099.0 aflplusplus 82800 20.0 2012.05 26.174465 1996.0 2002.50 2004.5 2006.25 2091.0 libafl 82800 20.0 1998.75 5.874790 1981.0 1998.75 2001.0 2002.00 2004.0 aflsmart 82800 20.0 1995.40 5.490662 1974.0 1994.00 1996.0 1998.25 2000.0 eclipser 82800 20.0 1990.45 11.250614 1972.0 1978.00 1994.0 1998.25 2003.0 afl 82800 20.0 1976.45 31.531897 1884.0 1971.75 1990.5 1995.25 2000.0 mopt 82800 20.0 1978.85 26.776413 1911.0 1975.00 1990.0 1993.50 2008.0 fairfuzz 82800 20.0 1984.10 13.814180 1945.0 1974.75 1988.5 1995.00 2000.0 libafl_fuzzbench_grimoire 82800 20.0 1983.50 10.221237 1973.0 1975.00 1979.5 1993.00 2003.0 aflfast 82800 20.0 1967.60 20.301413 1943.0 1948.25 1964.0 1989.50 1997.0
Vargha-Delaney A12 measure
The table summarizes the A12 values from the
pairwise Vargha-Delaney A measure of effect size.
Green cells indicate the probability the fuzzer in the
row will outperform the fuzzer in the column.
Mann-Whitney U test
The table summarizes the p values of
pairwise Mann-Whitney U tests.
Green cells indicate that the reached
coverage distribution of a given fuzzer pair
is significantly different.
-
Unique code coverage plots
Ranking by unique code branches covered
Each bar shows the total number of code branches found by a given fuzzer.
The colored area shows the number of unique code branches
(i.e., branches that were not covered by any other fuzzers).
Pairwise unique code coverage
Each cell represents the number of code branches covered by the fuzzer
of the column but not by the fuzzer of the row
-
Code coverage reports for each fuzzer on this benchmark
librdkafka_fuzz_regex summary
Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval
around the mean code coverage.
-
Sample statistics and statistical significance (code coverage)
Code coverage sample statistics
count mean std min 25% median 75% max fuzzer time libfuzzer 3600 10.0 379.8 4.661902 367.0 381.00 381.5 382.00 382.0 aflplusplus 3600 10.0 288.4 110.639756 0.0 266.25 325.0 350.00 371.0 afl 3600 10.0 169.6 184.713712 0.0 0.00 107.5 367.75 378.0 honggfuzz 3600 10.0 83.4 143.665661 0.0 0.00 0.0 114.75 344.0
Vargha-Delaney A12 measure
The table summarizes the A12 values from the
pairwise Vargha-Delaney A measure of effect size.
Green cells indicate the probability the fuzzer in the
row will outperform the fuzzer in the column.
Mann-Whitney U test
The table summarizes the p values of
pairwise Mann-Whitney U tests.
Green cells indicate that the reached
coverage distribution of a given fuzzer pair
is significantly different.
-
Unique code coverage plots
Ranking by unique code branches covered
Each bar shows the total number of code branches found by a given fuzzer.
The colored area shows the number of unique code branches
(i.e., branches that were not covered by any other fuzzers).
Pairwise unique code coverage
Each cell represents the number of code branches covered by the fuzzer
of the column but not by the fuzzer of the row
-
Code coverage reports for each fuzzer on this benchmark
libxml2_xml summary
Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval
around the mean code coverage.
-
Sample statistics and statistical significance (code coverage)
Code coverage sample statistics
count mean std min 25% median 75% max fuzzer time aflplusplus 82800 20.0 15738.10 52.114146 15640.0 15711.50 15724.5 15782.00 15822.0 libafl 82800 20.0 15592.55 43.236772 15461.0 15573.50 15592.5 15618.75 15655.0 honggfuzz 82800 20.0 15569.20 46.622335 15492.0 15536.00 15573.0 15586.00 15655.0 eclipser 82800 20.0 15423.45 79.669366 15257.0 15387.50 15423.5 15463.75 15601.0 libfuzzer 82800 20.0 15370.20 103.283259 15096.0 15311.25 15382.0 15427.75 15537.0 aflsmart 82800 20.0 15366.80 68.816767 15248.0 15322.50 15366.5 15410.75 15497.0 afl 82800 20.0 15346.60 76.391237 15154.0 15289.75 15353.5 15401.25 15482.0 libafl_fuzzbench_grimoire 82800 20.0 15240.05 281.312988 14500.0 15103.50 15336.0 15420.75 15530.0 mopt 82800 20.0 15318.60 62.704319 15155.0 15285.50 15326.0 15366.25 15397.0 aflfast 82800 20.0 15320.65 78.755635 15159.0 15271.75 15318.0 15356.00 15509.0 centipede 82800 20.0 14698.50 140.327249 14480.0 14610.75 14689.0 14786.25 14984.0 fairfuzz 82800 20.0 14146.65 259.273738 13616.0 14060.75 14144.0 14227.25 14834.0
Vargha-Delaney A12 measure
The table summarizes the A12 values from the
pairwise Vargha-Delaney A measure of effect size.
Green cells indicate the probability the fuzzer in the
row will outperform the fuzzer in the column.
Mann-Whitney U test
The table summarizes the p values of
pairwise Mann-Whitney U tests.
Green cells indicate that the reached
coverage distribution of a given fuzzer pair
is significantly different.
-
Unique code coverage plots
Ranking by unique code branches covered
Each bar shows the total number of code branches found by a given fuzzer.
The colored area shows the number of unique code branches
(i.e., branches that were not covered by any other fuzzers).
Pairwise unique code coverage
Each cell represents the number of code branches covered by the fuzzer
of the column but not by the fuzzer of the row
-
Code coverage reports for each fuzzer on this benchmark
libxslt_xpath summary
Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval
around the mean code coverage.
-
Sample statistics and statistical significance (code coverage)
Code coverage sample statistics
count mean std min 25% median 75% max fuzzer time aflplusplus 82800 20.0 11207.20 77.913988 11109.0 11143.25 11178.0 11268.00 11358.0 honggfuzz 82800 20.0 11045.10 108.722680 10855.0 10983.25 11050.5 11133.00 11226.0 libafl_fuzzbench_grimoire 82800 20.0 10999.00 85.604353 10877.0 10925.00 10987.0 11058.25 11152.0 libafl 82800 20.0 10930.35 68.430737 10825.0 10875.00 10953.0 10994.00 11018.0 fairfuzz 82800 20.0 10883.25 144.964197 10516.0 10827.75 10942.5 10959.25 11047.0 aflsmart 82800 20.0 10894.55 51.799690 10814.0 10859.25 10901.0 10915.00 11026.0 eclipser 82800 20.0 10895.65 65.938787 10729.0 10862.50 10897.5 10938.25 11010.0 afl 82800 20.0 10658.80 135.384676 10462.0 10529.00 10718.5 10770.25 10865.0 centipede 82800 20.0 10707.65 111.331782 10611.0 10631.75 10676.0 10724.25 11038.0 aflfast 82800 20.0 10636.05 92.919022 10363.0 10591.25 10649.5 10692.75 10786.0 mopt 82800 20.0 10571.50 126.561198 10325.0 10484.75 10580.0 10646.00 10781.0 libfuzzer 82800 20.0 10508.20 170.989258 10149.0 10417.50 10530.0 10632.75 10775.0
Vargha-Delaney A12 measure
The table summarizes the A12 values from the
pairwise Vargha-Delaney A measure of effect size.
Green cells indicate the probability the fuzzer in the
row will outperform the fuzzer in the column.
Mann-Whitney U test
The table summarizes the p values of
pairwise Mann-Whitney U tests.
Green cells indicate that the reached
coverage distribution of a given fuzzer pair
is significantly different.
-
Unique code coverage plots
Ranking by unique code branches covered
Each bar shows the total number of code branches found by a given fuzzer.
The colored area shows the number of unique code branches
(i.e., branches that were not covered by any other fuzzers).
Pairwise unique code coverage
Each cell represents the number of code branches covered by the fuzzer
of the column but not by the fuzzer of the row
-
Code coverage reports for each fuzzer on this benchmark
mbedtls_fuzz_dtlsclient summary
Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval
around the mean code coverage.
-
Sample statistics and statistical significance (code coverage)
Code coverage sample statistics
count mean std min 25% median 75% max fuzzer time libafl 82800 20.0 3259.75 386.054383 2723.0 2744.25 3327.5 3660.75 3694.0 libafl_fuzzbench_grimoire 82800 20.0 3263.65 467.193894 2732.0 2771.00 3241.5 3720.25 3889.0 aflplusplus 82800 20.0 2793.15 33.280744 2723.0 2769.25 2792.0 2819.50 2856.0 fairfuzz 82800 20.0 2820.70 203.472771 2730.0 2753.25 2779.0 2794.75 3676.0 eclipser 82800 20.0 2774.45 217.678052 2702.0 2715.25 2726.0 2733.25 3697.0 aflsmart 82800 20.0 2722.05 16.750412 2697.0 2712.50 2720.0 2728.25 2771.0 libfuzzer 82800 20.0 2725.50 145.014518 2654.0 2675.00 2692.5 2709.50 3331.0 afl 82800 20.0 2695.70 25.393949 2653.0 2680.75 2692.0 2716.50 2744.0 mopt 82800 20.0 2644.60 88.620658 2444.0 2562.50 2688.0 2699.50 2756.0 honggfuzz 82800 20.0 2695.50 41.359146 2647.0 2670.00 2681.0 2704.50 2802.0 centipede 82800 20.0 2639.90 12.531644 2621.0 2630.50 2639.0 2644.50 2673.0 aflfast 82800 20.0 2638.30 14.026666 2607.0 2630.75 2638.0 2643.50 2675.0
Vargha-Delaney A12 measure
The table summarizes the A12 values from the
pairwise Vargha-Delaney A measure of effect size.
Green cells indicate the probability the fuzzer in the
row will outperform the fuzzer in the column.
Mann-Whitney U test
The table summarizes the p values of
pairwise Mann-Whitney U tests.
Green cells indicate that the reached
coverage distribution of a given fuzzer pair
is significantly different.
-
Unique code coverage plots
Ranking by unique code branches covered
Each bar shows the total number of code branches found by a given fuzzer.
The colored area shows the number of unique code branches
(i.e., branches that were not covered by any other fuzzers).
Pairwise unique code coverage
Each cell represents the number of code branches covered by the fuzzer
of the column but not by the fuzzer of the row
-
Code coverage reports for each fuzzer on this benchmark
openh264_decoder_fuzzer summary
Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval
around the mean code coverage.
-
Sample statistics and statistical significance (code coverage)
Code coverage sample statistics
count mean std min 25% median 75% max fuzzer time honggfuzz 82800 20.0 9546.55 6.549206 9535.0 9543.75 9545.0 9549.00 9559.0 mopt 82800 20.0 9539.10 13.540971 9517.0 9525.75 9540.5 9548.25 9563.0 libafl 82800 20.0 9536.75 15.095747 9482.0 9533.50 9539.5 9542.75 9553.0 aflsmart 82800 20.0 9518.75 42.213586 9433.0 9511.75 9533.5 9543.00 9582.0 aflplusplus 82800 20.0 9535.40 9.075125 9522.0 9528.75 9532.5 9544.00 9549.0 eclipser 82800 20.0 9518.45 26.296838 9441.0 9515.25 9526.5 9533.75 9541.0 aflfast 82800 20.0 9509.10 52.625689 9380.0 9493.00 9525.5 9551.00 9561.0 afl 82800 20.0 9511.50 35.952454 9444.0 9500.75 9524.5 9535.25 9555.0 libfuzzer 82800 20.0 9424.80 48.364406 9335.0 9394.00 9420.5 9455.00 9522.0 libafl_fuzzbench_grimoire 82800 20.0 9348.10 88.981902 9103.0 9316.00 9331.0 9382.00 9511.0 fairfuzz 82800 20.0 8690.75 181.798001 8400.0 8579.25 8713.0 8780.50 9055.0
Vargha-Delaney A12 measure
The table summarizes the A12 values from the
pairwise Vargha-Delaney A measure of effect size.
Green cells indicate the probability the fuzzer in the
row will outperform the fuzzer in the column.
Mann-Whitney U test
The table summarizes the p values of
pairwise Mann-Whitney U tests.
Green cells indicate that the reached
coverage distribution of a given fuzzer pair
is significantly different.
-
Unique code coverage plots
Ranking by unique code branches covered
Each bar shows the total number of code branches found by a given fuzzer.
The colored area shows the number of unique code branches
(i.e., branches that were not covered by any other fuzzers).
Pairwise unique code coverage
Each cell represents the number of code branches covered by the fuzzer
of the column but not by the fuzzer of the row
-
Code coverage reports for each fuzzer on this benchmark
openssl_x509 summary
Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval
around the mean code coverage.
-
Sample statistics and statistical significance (code coverage)
Code coverage sample statistics
count mean std min 25% median 75% max fuzzer time libafl_fuzzbench_grimoire 82800 20.0 5831.25 1.585294 5827.0 5831.00 5831.0 5833.00 5833.0 aflsmart 82800 20.0 5830.35 2.539685 5827.0 5829.00 5830.0 5831.00 5840.0 eclipser 82800 20.0 5829.45 3.119970 5817.0 5829.00 5830.0 5831.00 5832.0 libfuzzer 82800 20.0 5827.20 5.615767 5817.0 5822.50 5829.0 5832.00 5833.0 mopt 82800 20.0 5824.90 6.290343 5810.0 5823.00 5828.0 5829.00 5829.0 afl 82800 20.0 5826.20 4.323741 5817.0 5824.25 5827.5 5829.25 5831.0 aflplusplus 82800 20.0 5823.55 6.863136 5807.0 5821.00 5825.0 5827.50 5834.0 centipede 82800 20.0 5823.80 5.863356 5810.0 5822.00 5825.0 5828.00 5833.0 libafl 82800 20.0 5826.20 3.968295 5821.0 5823.00 5825.0 5831.00 5831.0 fairfuzz 82800 20.0 5821.50 2.874571 5817.0 5819.00 5822.0 5823.00 5827.0 aflfast 82800 20.0 5819.20 6.304551 5804.0 5816.75 5820.5 5824.00 5827.0 honggfuzz 82800 20.0 5817.45 7.917170 5806.0 5810.00 5819.5 5822.00 5834.0
Vargha-Delaney A12 measure
The table summarizes the A12 values from the
pairwise Vargha-Delaney A measure of effect size.
Green cells indicate the probability the fuzzer in the
row will outperform the fuzzer in the column.
Mann-Whitney U test
The table summarizes the p values of
pairwise Mann-Whitney U tests.
Green cells indicate that the reached
coverage distribution of a given fuzzer pair
is significantly different.
-
Unique code coverage plots
Ranking by unique code branches covered
Each bar shows the total number of code branches found by a given fuzzer.
The colored area shows the number of unique code branches
(i.e., branches that were not covered by any other fuzzers).
Pairwise unique code coverage
Each cell represents the number of code branches covered by the fuzzer
of the column but not by the fuzzer of the row
-
Code coverage reports for each fuzzer on this benchmark
openthread_ot-ip6-send-fuzzer summary
Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval
around the mean code coverage.
-
Sample statistics and statistical significance (code coverage)
Code coverage sample statistics
count mean std min 25% median 75% max fuzzer time libafl 82800 20.0 3321.15 261.997243 2908.0 3038.00 3456.5 3534.25 3581.0 libafl_fuzzbench_grimoire 82800 20.0 3360.35 310.061331 3012.0 3060.75 3418.0 3525.75 4160.0 libfuzzer 82800 20.0 3079.40 13.287905 3053.0 3070.00 3078.0 3090.00 3108.0 aflplusplus 82800 20.0 3106.30 135.384870 3042.0 3053.75 3065.0 3075.50 3524.0 eclipser 82800 20.0 3022.20 46.183387 2911.0 3010.75 3038.5 3046.75 3087.0 honggfuzz 82800 20.0 3067.55 249.356957 2896.0 2918.75 3037.5 3047.50 3855.0 mopt 82800 20.0 2945.55 143.939854 2807.0 2902.25 2914.0 2923.75 3490.0 aflsmart 82800 20.0 2889.60 38.982317 2826.0 2836.50 2909.5 2913.25 2934.0 afl 82800 20.0 2889.20 52.609385 2819.0 2831.25 2904.5 2914.50 3006.0 aflfast 82800 20.0 2877.30 39.174508 2824.0 2830.75 2892.5 2908.75 2927.0 centipede 82800 20.0 2849.85 60.583370 2728.0 2787.00 2876.0 2896.75 2925.0 fairfuzz 82800 20.0 2756.10 75.288847 2628.0 2724.25 2751.5 2788.50 2907.0
Vargha-Delaney A12 measure
The table summarizes the A12 values from the
pairwise Vargha-Delaney A measure of effect size.
Green cells indicate the probability the fuzzer in the
row will outperform the fuzzer in the column.
Mann-Whitney U test
The table summarizes the p values of
pairwise Mann-Whitney U tests.
Green cells indicate that the reached
coverage distribution of a given fuzzer pair
is significantly different.
-
Unique code coverage plots
Ranking by unique code branches covered
Each bar shows the total number of code branches found by a given fuzzer.
The colored area shows the number of unique code branches
(i.e., branches that were not covered by any other fuzzers).
Pairwise unique code coverage
Each cell represents the number of code branches covered by the fuzzer
of the column but not by the fuzzer of the row
-
Code coverage reports for each fuzzer on this benchmark
proj4_proj_crs_to_crs_fuzzer summary
Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval
around the mean code coverage.
-
Sample statistics and statistical significance (code coverage)
Code coverage sample statistics
count mean std min 25% median 75% max fuzzer time honggfuzz 82800 20.0 7932.25 219.503956 7504.0 7859.00 8002.0 8059.00 8245.0 libafl_fuzzbench_grimoire 82800 20.0 7698.35 97.505344 7473.0 7626.25 7720.0 7761.75 7873.0 libfuzzer 82800 20.0 7556.85 86.613038 7414.0 7485.75 7543.5 7608.50 7727.0 aflplusplus 82800 20.0 7213.55 197.000260 6897.0 7077.00 7247.5 7343.75 7639.0 eclipser 82800 20.0 4890.40 462.103247 3876.0 4703.25 4924.5 5103.75 5696.0 centipede 82800 20.0 839.80 73.863744 812.0 820.75 824.0 826.50 1153.0 libafl 82800 20.0 809.50 12.601170 763.0 806.50 808.5 817.00 826.0 fairfuzz 82800 20.0 780.45 74.632768 497.0 747.75 807.5 818.25 826.0 afl 82800 20.0 802.20 21.060314 740.0 804.00 807.0 809.75 819.0 aflsmart 82800 20.0 784.00 73.153767 497.0 805.00 806.5 818.00 819.0 mopt 82800 20.0 770.30 74.056948 497.0 740.75 806.5 815.00 820.0 aflfast 82800 20.0 738.30 115.000275 475.0 731.75 794.5 806.50 817.0
Vargha-Delaney A12 measure
The table summarizes the A12 values from the
pairwise Vargha-Delaney A measure of effect size.
Green cells indicate the probability the fuzzer in the
row will outperform the fuzzer in the column.
Mann-Whitney U test
The table summarizes the p values of
pairwise Mann-Whitney U tests.
Green cells indicate that the reached
coverage distribution of a given fuzzer pair
is significantly different.
-
Unique code coverage plots
Ranking by unique code branches covered
Each bar shows the total number of code branches found by a given fuzzer.
The colored area shows the number of unique code branches
(i.e., branches that were not covered by any other fuzzers).
Pairwise unique code coverage
Each cell represents the number of code branches covered by the fuzzer
of the column but not by the fuzzer of the row
-
Code coverage reports for each fuzzer on this benchmark
re2_fuzzer summary
Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval
around the mean code coverage.
-
Sample statistics and statistical significance (code coverage)
Code coverage sample statistics
count mean std min 25% median 75% max fuzzer time libfuzzer 82800 20.0 2884.00 2.492093 2880.0 2882.00 2883.5 2885.00 2891.0 aflplusplus 82800 20.0 2877.80 4.926833 2866.0 2876.00 2878.0 2880.25 2885.0 aflsmart 82800 20.0 2868.55 4.936172 2860.0 2865.75 2868.5 2872.50 2876.0 libafl_fuzzbench_grimoire 82800 20.0 2867.70 7.484018 2850.0 2864.75 2868.5 2871.25 2885.0 eclipser 82800 20.0 2860.85 23.079439 2785.0 2861.50 2867.5 2872.25 2878.0 aflfast 82800 20.0 2865.45 6.270021 2849.0 2864.50 2866.5 2870.00 2873.0 fairfuzz 82800 20.0 2834.65 73.055332 2631.0 2856.25 2864.0 2869.00 2875.0 afl 82800 20.0 2851.80 33.783521 2758.0 2857.75 2863.5 2869.25 2873.0 mopt 82800 20.0 2856.30 7.947856 2843.0 2849.75 2858.5 2862.25 2869.0 libafl 82800 20.0 2856.85 7.761409 2846.0 2849.00 2857.5 2864.50 2868.0 honggfuzz 82800 20.0 2851.85 8.963523 2836.0 2846.50 2850.0 2856.25 2869.0 centipede 82800 20.0 2772.60 17.005881 2736.0 2764.75 2772.5 2785.00 2812.0
Vargha-Delaney A12 measure
The table summarizes the A12 values from the
pairwise Vargha-Delaney A measure of effect size.
Green cells indicate the probability the fuzzer in the
row will outperform the fuzzer in the column.
Mann-Whitney U test
The table summarizes the p values of
pairwise Mann-Whitney U tests.
Green cells indicate that the reached
coverage distribution of a given fuzzer pair
is significantly different.
-
Unique code coverage plots
Ranking by unique code branches covered
Each bar shows the total number of code branches found by a given fuzzer.
The colored area shows the number of unique code branches
(i.e., branches that were not covered by any other fuzzers).
Pairwise unique code coverage
Each cell represents the number of code branches covered by the fuzzer
of the column but not by the fuzzer of the row
-
Code coverage reports for each fuzzer on this benchmark
sqlite3_ossfuzz summary
Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval
around the mean code coverage.
-
Sample statistics and statistical significance (code coverage)
Code coverage sample statistics
count mean std min 25% median 75% max fuzzer time aflsmart 82800 20.0 19264.550000 284.391590 18652.0 19128.25 19284.0 19399.00 19883.0 eclipser 82800 20.0 19215.250000 317.380707 18729.0 18953.50 19191.5 19428.00 19939.0 mopt 82800 20.0 19038.100000 288.714191 18474.0 18934.50 19060.5 19248.75 19483.0 afl 82800 20.0 18945.550000 270.730294 18414.0 18732.25 18885.5 19180.25 19380.0 libafl 82800 20.0 18853.950000 124.714189 18638.0 18775.75 18835.0 18946.75 19056.0 aflfast 82800 20.0 18711.000000 281.495069 18104.0 18609.25 18713.5 18918.25 19183.0 aflplusplus 82800 20.0 18666.050000 1523.443795 15066.0 17937.50 18494.0 20214.25 20568.0 libafl_fuzzbench_grimoire 82800 20.0 16429.500000 959.512678 13848.0 15576.25 16908.0 17060.00 17430.0 libfuzzer 82800 20.0 16541.150000 377.282426 15825.0 16287.00 16548.5 16731.00 17535.0 honggfuzz 82800 20.0 14800.300000 476.602019 13799.0 14555.75 14765.0 15077.25 15724.0 fairfuzz 82800 20.0 13246.650000 1692.282646 10991.0 11657.25 13522.5 14776.25 16178.0 centipede 82800 18.0 13391.333333 312.381554 12856.0 13199.75 13329.0 13560.00 13974.0
Vargha-Delaney A12 measure
The table summarizes the A12 values from the
pairwise Vargha-Delaney A measure of effect size.
Green cells indicate the probability the fuzzer in the
row will outperform the fuzzer in the column.
Mann-Whitney U test
The table summarizes the p values of
pairwise Mann-Whitney U tests.
Green cells indicate that the reached
coverage distribution of a given fuzzer pair
is significantly different.
-
Unique code coverage plots
Ranking by unique code branches covered
Each bar shows the total number of code branches found by a given fuzzer.
The colored area shows the number of unique code branches
(i.e., branches that were not covered by any other fuzzers).
Pairwise unique code coverage
Each cell represents the number of code branches covered by the fuzzer
of the column but not by the fuzzer of the row
-
Code coverage reports for each fuzzer on this benchmark
stb_stbi_read_fuzzer summary
Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval
around the mean code coverage.
-
Sample statistics and statistical significance (code coverage)
Code coverage sample statistics
count mean std min 25% median 75% max fuzzer time aflplusplus 82800 20.0 2170.25 55.177107 2112.0 2116.00 2194.5 2212.25 2299.0 libafl_fuzzbench_grimoire 82800 20.0 2196.50 30.663711 2151.0 2188.00 2192.5 2196.75 2285.0 libafl 82800 20.0 2152.45 42.371136 2078.0 2108.50 2178.0 2189.25 2195.0 honggfuzz 82800 20.0 2129.60 34.347144 2106.0 2112.00 2114.5 2117.25 2199.0 eclipser 82800 20.0 2109.45 30.165290 2080.0 2098.50 2106.0 2109.25 2192.0 libfuzzer 82800 20.0 2036.15 51.490699 1988.0 2005.50 2014.0 2079.00 2183.0 afl 82800 20.0 2025.70 53.595070 1982.0 2001.75 2005.5 2010.25 2192.0 aflsmart 82800 20.0 2016.00 29.568831 1983.0 2004.00 2005.5 2008.00 2090.0 mopt 82800 20.0 2003.60 38.368573 1953.0 1985.50 1994.0 2004.25 2088.0 aflfast 82800 20.0 1983.90 30.450823 1903.0 1974.00 1984.0 2000.25 2065.0 fairfuzz 82800 20.0 1967.25 49.496278 1854.0 1938.50 1983.0 1996.50 2086.0 centipede 82800 20.0 1959.55 4.547700 1952.0 1956.75 1958.5 1962.75 1967.0
Vargha-Delaney A12 measure
The table summarizes the A12 values from the
pairwise Vargha-Delaney A measure of effect size.
Green cells indicate the probability the fuzzer in the
row will outperform the fuzzer in the column.
Mann-Whitney U test
The table summarizes the p values of
pairwise Mann-Whitney U tests.
Green cells indicate that the reached
coverage distribution of a given fuzzer pair
is significantly different.
-
Unique code coverage plots
Ranking by unique code branches covered
Each bar shows the total number of code branches found by a given fuzzer.
The colored area shows the number of unique code branches
(i.e., branches that were not covered by any other fuzzers).
Pairwise unique code coverage
Each cell represents the number of code branches covered by the fuzzer
of the column but not by the fuzzer of the row
-
Code coverage reports for each fuzzer on this benchmark
systemd_fuzz-link-parser summary
Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval
around the mean code coverage.
-
Sample statistics and statistical significance (code coverage)
Code coverage sample statistics
count mean std min 25% median 75% max fuzzer time aflplusplus 82800 20.0 234.25 6.640189 223.0 226.0 239.0 239.00 240.0 libafl_fuzzbench_grimoire 82800 20.0 237.50 1.100239 235.0 237.0 237.0 239.00 239.0 centipede 82800 20.0 234.95 0.223607 234.0 235.0 235.0 235.00 235.0 honggfuzz 82800 20.0 233.45 3.440854 224.0 233.0 233.0 237.00 238.0 libafl 82800 20.0 223.55 1.394538 221.0 223.0 224.0 224.00 226.0 eclipser 82800 20.0 220.65 1.496487 219.0 219.0 221.0 222.00 223.0 afl 82800 20.0 220.20 1.609184 217.0 219.0 219.5 222.00 223.0 aflfast 82800 20.0 220.35 1.496487 219.0 219.0 219.5 222.00 223.0 aflsmart 82800 20.0 220.10 1.586124 217.0 219.0 219.0 221.25 223.0 mopt 82800 20.0 220.20 1.542384 219.0 219.0 219.0 222.00 223.0 fairfuzz 82800 20.0 208.85 9.653333 189.0 202.0 206.0 219.00 222.0 libfuzzer 82800 20.0 198.45 37.134854 156.0 156.0 193.5 238.00 238.0
Vargha-Delaney A12 measure
The table summarizes the A12 values from the
pairwise Vargha-Delaney A measure of effect size.
Green cells indicate the probability the fuzzer in the
row will outperform the fuzzer in the column.
Mann-Whitney U test
The table summarizes the p values of
pairwise Mann-Whitney U tests.
Green cells indicate that the reached
coverage distribution of a given fuzzer pair
is significantly different.
-
Unique code coverage plots
Ranking by unique code branches covered
Each bar shows the total number of code branches found by a given fuzzer.
The colored area shows the number of unique code branches
(i.e., branches that were not covered by any other fuzzers).
Pairwise unique code coverage
Each cell represents the number of code branches covered by the fuzzer
of the column but not by the fuzzer of the row
-
Code coverage reports for each fuzzer on this benchmark
vorbis_decode_fuzzer summary
Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval
around the mean code coverage.
-
Sample statistics and statistical significance (code coverage)
Code coverage sample statistics
count mean std min 25% median 75% max fuzzer time libfuzzer 82800 20.0 1272.00 16.270736 1264.0 1266.00 1267.0 1269.00 1336.0 aflplusplus 82800 20.0 1266.35 2.996050 1259.0 1265.00 1266.0 1268.25 1271.0 aflsmart 82800 20.0 1255.00 2.919986 1250.0 1253.00 1254.5 1257.25 1262.0 mopt 82800 20.0 1250.20 12.722132 1199.0 1250.00 1254.0 1255.25 1258.0 afl 82800 20.0 1252.50 3.831998 1244.0 1250.50 1252.0 1255.25 1258.0 aflfast 82800 20.0 1246.80 15.340349 1196.0 1248.50 1252.0 1254.00 1256.0 eclipser 82800 20.0 1251.95 4.358597 1244.0 1249.75 1252.0 1255.25 1259.0 libafl 82800 20.0 1251.35 3.513508 1247.0 1249.00 1251.0 1252.00 1261.0 fairfuzz 82800 20.0 1234.70 28.252806 1160.0 1222.50 1249.5 1253.50 1258.0 honggfuzz 82800 20.0 1249.40 4.805698 1235.0 1248.00 1249.0 1253.00 1258.0 libafl_fuzzbench_grimoire 82800 20.0 1249.10 4.789242 1237.0 1246.75 1249.0 1252.00 1259.0 centipede 82800 20.0 1149.15 17.263515 1122.0 1138.50 1145.5 1161.75 1181.0
Vargha-Delaney A12 measure
The table summarizes the A12 values from the
pairwise Vargha-Delaney A measure of effect size.
Green cells indicate the probability the fuzzer in the
row will outperform the fuzzer in the column.
Mann-Whitney U test
The table summarizes the p values of
pairwise Mann-Whitney U tests.
Green cells indicate that the reached
coverage distribution of a given fuzzer pair
is significantly different.
-
Unique code coverage plots
Ranking by unique code branches covered
Each bar shows the total number of code branches found by a given fuzzer.
The colored area shows the number of unique code branches
(i.e., branches that were not covered by any other fuzzers).
Pairwise unique code coverage
Each cell represents the number of code branches covered by the fuzzer
of the column but not by the fuzzer of the row
-
Code coverage reports for each fuzzer on this benchmark
woff2_convert_woff2ttf_fuzzer summary
Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval
around the mean code coverage.
-
Sample statistics and statistical significance (code coverage)
Code coverage sample statistics
count mean std min 25% median 75% max fuzzer time libafl 82800 20.0 1174.10 13.364919 1142.0 1167.75 1174.0 1182.50 1194.0 aflplusplus 82800 20.0 1176.65 12.261514 1153.0 1170.25 1173.5 1184.50 1200.0 honggfuzz 82800 20.0 1165.80 25.071687 1106.0 1159.75 1172.0 1183.00 1196.0 libfuzzer 82800 20.0 1135.50 65.774335 1002.0 1097.00 1139.0 1195.00 1214.0 aflsmart 82800 20.0 1133.60 7.036746 1116.0 1130.75 1134.5 1138.25 1144.0 libafl_fuzzbench_grimoire 82800 20.0 1123.75 21.390787 1088.0 1113.00 1126.5 1138.75 1156.0 afl 82800 20.0 1114.60 21.315673 1072.0 1103.50 1119.5 1131.00 1145.0 mopt 82800 20.0 1114.45 16.040984 1089.0 1102.50 1116.0 1128.25 1136.0 aflfast 82800 20.0 1083.30 26.647998 1025.0 1076.00 1089.5 1101.50 1114.0 centipede 82800 20.0 1076.80 12.033287 1056.0 1070.25 1073.0 1085.75 1099.0 fairfuzz 82800 20.0 996.00 26.789629 966.0 983.00 988.5 998.50 1080.0
Vargha-Delaney A12 measure
The table summarizes the A12 values from the
pairwise Vargha-Delaney A measure of effect size.
Green cells indicate the probability the fuzzer in the
row will outperform the fuzzer in the column.
Mann-Whitney U test
The table summarizes the p values of
pairwise Mann-Whitney U tests.
Green cells indicate that the reached
coverage distribution of a given fuzzer pair
is significantly different.
-
Unique code coverage plots
Ranking by unique code branches covered
Each bar shows the total number of code branches found by a given fuzzer.
The colored area shows the number of unique code branches
(i.e., branches that were not covered by any other fuzzers).
Pairwise unique code coverage
Each cell represents the number of code branches covered by the fuzzer
of the column but not by the fuzzer of the row
-
Code coverage reports for each fuzzer on this benchmark
zlib_zlib_uncompress_fuzzer summary
Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval
around the mean code coverage.
-
Sample statistics and statistical significance (code coverage)
Code coverage sample statistics
count mean std min 25% median 75% max fuzzer time libfuzzer 82800 20.0 468.65 4.487116 462.0 463.00 472.0 472.00 472.0 aflplusplus 82800 20.0 462.75 5.580747 456.0 459.75 460.0 468.50 472.0 fairfuzz 82800 20.0 460.20 4.162489 455.0 457.50 460.0 461.25 469.0 honggfuzz 82800 20.0 460.15 3.513508 456.0 458.00 459.0 461.25 471.0 aflsmart 82800 20.0 458.35 3.082634 455.0 456.00 458.5 459.25 468.0 afl 82800 20.0 457.10 3.210181 449.0 455.75 458.0 459.25 461.0 eclipser 82800 20.0 456.40 7.556106 427.0 455.75 458.0 460.25 462.0 mopt 82800 20.0 454.15 9.804805 419.0 453.75 455.0 458.50 468.0 centipede 82800 20.0 454.50 3.804430 451.0 451.00 454.0 457.00 464.0 aflfast 82800 20.0 451.05 5.266328 437.0 448.75 451.0 454.25 459.0 libafl_fuzzbench_grimoire 82800 20.0 450.10 6.576673 440.0 445.00 450.0 453.50 462.0 libafl 82800 20.0 450.35 5.153180 443.0 448.00 448.5 451.25 460.0
Vargha-Delaney A12 measure
The table summarizes the A12 values from the
pairwise Vargha-Delaney A measure of effect size.
Green cells indicate the probability the fuzzer in the
row will outperform the fuzzer in the column.
Mann-Whitney U test
The table summarizes the p values of
pairwise Mann-Whitney U tests.
Green cells indicate that the reached
coverage distribution of a given fuzzer pair
is significantly different.
-
Unique code coverage plots
Ranking by unique code branches covered
Each bar shows the total number of code branches found by a given fuzzer.
The colored area shows the number of unique code branches
(i.e., branches that were not covered by any other fuzzers).
Pairwise unique code coverage
Each cell represents the number of code branches covered by the fuzzer
of the column but not by the fuzzer of the row
-
Code coverage reports for each fuzzer on this benchmark
experiment data
You can download the raw data for this report here.Check out the documentation on how to create customized reports using this data. Also see some example Colab notebooks for doing custom analysis on the data here.
Experiment Description:
(None,)