FuzzBench: gen-cov-test-4-3 report

Please consider this as a preliminary report to demonstrate the capabilities of FuzzBench. While we have tried our best, we have not confirmed that we configured everything correctly. We are hoping to work together with the community to validate results and improve the set of fuzzers, benchmarks, and their configurations in the future. See FAQ for more details.

experiment summary

We show two different aggregate (cross-benchmark) rankings of fuzzers. The first is based on the average of per-benchmarks scores, where the score represents the percentage of the highest reached median coverage on a given benchmark (higher value is better). The second ranking shows the average rank of fuzzers, after we rank them on each benchmark according to their median reached covereges (lower value is better).

By avg. score

	average normalized score
fuzzer
afl	100.00
libfuzzer	91.92

By avg. rank

	average rank
fuzzer
afl	1.0
libfuzzer	2.0

Critical difference diagram

The diagram visualizes the average rank of fuzzers (second ranking above) while showing the significance of the differences as well. What is considered a "critical difference" (CD) is based on the Friedman/Nemenyi post-hoc test. See more in the documentation.

Note: If a fuzzer does not support all benchmarks, its ranking as shown in this diagram can be lower than it should be. So please check the list of supported benchmarks for the fuzzer(s) of your interest. The list could be specified in the fuzzer's README.md like this.

Median coverages on each benchmark

fuzzer	afl	libfuzzer
benchmark
bloaty_fuzz_target	7516.0	6334.5
libpng-1.2.56	1509.0	1502.5

bloaty_fuzz_target summary

Ranking by median reached coverage

Reached coverage distribution

Mean coverage growth over time

* The error bands show the 95% confidence interval around the mean coverage.

Sample statistics and statistical significance

Coverage sample statistics

		count	mean	std	min	25%	median	75%	max
fuzzer	time
afl	1800	2.0	7516.0	547.300649	7129.0	7322.50	7516.0	7709.50	7903.0
libfuzzer	1800	2.0	6334.5	279.307179	6137.0	6235.75	6334.5	6433.25	6532.0

Mann-Whitney U test

The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

Coverage reports for each fuzzer on this benchmark

libfuzzer

afl

libpng-1.2.56 summary