FuzzBench: 2023-12-10-standard-bug report

experiment summary

We show two different aggregate (cross-benchmark) rankings of fuzzers. The first is based on the average of per-benchmarks scores, where the score represents the percentage of the highest reached median bug-coverage on a given benchmark (higher value is better). The second ranking shows the average rank of fuzzers, after we rank them on each benchmark according to their median reached bug-covereges (lower value is better).
By avg. score
average normalized score
fuzzer
afl 75.0
aflfast 75.0
aflsmart 75.0
eclipser 75.0
libafl 75.0
mopt 75.0
aflplusplus 50.0
centipede 50.0
fairfuzz 50.0
libfuzzer 50.0
honggfuzz 25.0
By avg. rank
average rank
fuzzer
afl 1.2
aflfast 1.2
aflsmart 1.2
eclipser 1.2
libafl 1.2
mopt 1.2
aflplusplus 2.6
libfuzzer 2.6
fairfuzz 2.8
centipede 4.0
honggfuzz 4.2
  • Critical difference diagram
    The diagram visualizes the average rank of fuzzers (second ranking above) while showing the significance of the differences as well. What is considered a "critical difference" (CD) is based on the Friedman/Nemenyi post-hoc test. See more in the documentation.
    Note: If a fuzzer does not support all benchmarks, its ranking as shown in this diagram can be lower than it should be. So please check the list of supported benchmarks for the fuzzer(s) of your interest. The list could be specified in the fuzzer's README.md like this.
  • Median relative code-coverages on each benchmark

    Note: The relative coverage summary table shows the median relative performance of each fuzzer to the experiment maximum. Thus the highest relative performance may not be 100%.
    trial_relative_coverage = trial_coverage / experiment_max_coverage

      libafl mopt afl eclipser aflsmart aflfast aflplusplus honggfuzz libfuzzer fairfuzz centipede
    FuzzerMedian 95.00 96.00 95.00 94.00 94.00 93.00 92.00 92.00 88.00 89.00 84.00
    FuzzerMean 93.40 90.40 90.20 90.00 89.80 89.00 88.80 87.20 85.20 85.00 83.25
    bloaty_fuzz_target_52948c 95.00 96.00 95.00 94.00 94.00 93.00 90.00 92.00 88.00 80.00 nan
    harfbuzz_hb-shape-fuzzer_17863b 99.00 96.00 96.00 96.00 96.00 95.00 95.00 96.00 94.00 89.00 87.00
    libxml2_xml_e85b9b 99.00 98.00 98.00 98.00 98.00 97.00 96.00 85.00 82.00 96.00 81.00
    mbedtls_fuzz_dtlsclient_7c6b0e 82.00 71.00 71.00 71.00 70.00 69.00 71.00 70.00 70.00 71.00 69.00
    php_php-fuzz-parser_0dbedb 92.00 91.00 91.00 91.00 91.00 91.00 92.00 93.00 92.00 89.00 96.00
    • Fuzzers are sorted by "FuzzerMean" (average median relative coverage), highest on the left.
    • Green background = highest relative median coverage.
    • Blue gradient background = greater than 95% relative median coverage.
  • Median relative bug-coverages on each benchmark

    Note: The relative coverage summary table shows the median relative performance of each fuzzer to the experiment maximum. Thus the highest relative performance may not be 100%.
    trial_relative_coverage = trial_coverage / experiment_max_coverage

      afl aflfast aflsmart eclipser libafl mopt aflplusplus fairfuzz libfuzzer centipede honggfuzz
    FuzzerMedian 100.00 100.00 100.00 100.00 100.00 100.00 0.00 0.00 0.00 25.00 0.00
    FuzzerMean 60.00 60.00 60.00 60.00 60.00 60.00 40.00 40.00 40.00 37.50 20.00
    bloaty_fuzz_target_52948c 100.00 100.00 100.00 100.00 100.00 100.00 100.00 0.00 100.00 nan 0.00
    harfbuzz_hb-shape-fuzzer_17863b 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00
    libxml2_xml_e85b9b 100.00 100.00 100.00 100.00 100.00 100.00 0.00 100.00 0.00 0.00 0.00
    mbedtls_fuzz_dtlsclient_7c6b0e 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
    php_php-fuzz-parser_0dbedb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 50.00 0.00
    • Fuzzers are sorted by "FuzzerMean" (average median relative coverage), highest on the left.
    • Green background = highest relative median coverage.
    • Blue gradient background = greater than 95% relative median coverage.
  • Total unique bugs found on each benchmark
      Total libafl libfuzzer afl aflsmart eclipser mopt aflfast aflplusplus centipede fairfuzz honggfuzz
    FuzzerSum 13 10 10 9 9 9 9 8 8 8 7 7
    bloaty_fuzz_target_52948c 1 1 1 1 1 1 1 1 1 0 1 1
    harfbuzz_hb-shape-fuzzer_17863b 7 6 6 6 5 4 5 5 4 3 4 3
    libxml2_xml_e85b9b 2 2 2 2 2 2 2 2 1 2 2 2
    mbedtls_fuzz_dtlsclient_7c6b0e 0 0 0 0 0 0 0 0 0 0 0 0
    php_php-fuzz-parser_0dbedb 3 1 1 0 1 2 1 0 2 3 0 1
    • Fuzzers are sorted by "FuzzerSum", highest on the left.
    • Green background = most unique bugs found.
    • *note: This table represents unique bugs found across all trials.

bloaty_fuzz_target_52948c summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
error
The following fuzzers do not have enough samples: aflplusplus.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 1.000000 0.000000 1.0 1.0 1.0 1.0 1.0
    aflfast 82800 20.0 0.950000 0.223607 0.0 1.0 1.0 1.0 1.0
    aflplusplus 82800 13.0 0.846154 0.375534 0.0 1.0 1.0 1.0 1.0
    aflsmart 82800 20.0 1.000000 0.000000 1.0 1.0 1.0 1.0 1.0
    eclipser 82800 20.0 1.000000 0.000000 1.0 1.0 1.0 1.0 1.0
    libafl 82800 20.0 0.900000 0.307794 0.0 1.0 1.0 1.0 1.0
    libfuzzer 82800 20.0 0.900000 0.307794 0.0 1.0 1.0 1.0 1.0
    mopt 82800 20.0 1.000000 0.000000 1.0 1.0 1.0 1.0 1.0
    fairfuzz 82800 20.0 0.050000 0.223607 0.0 0.0 0.0 0.0 1.0
    honggfuzz 82800 20.0 0.350000 0.489360 0.0 0.0 0.0 1.0 1.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    mopt 82800 20.0 6011.800000 113.852906 5855.0 5912.50 6019.0 6091.50 6235.0
    afl 82800 20.0 5888.550000 114.767717 5690.0 5794.00 5937.0 5977.50 6069.0
    libafl 82800 20.0 5703.650000 469.483285 4372.0 5485.00 5936.0 6046.75 6140.0
    aflsmart 82800 20.0 5911.050000 79.416606 5766.0 5856.50 5918.5 5974.75 6033.0
    eclipser 82800 20.0 5867.800000 114.166822 5655.0 5769.25 5901.0 5963.50 6002.0
    aflfast 82800 20.0 5833.200000 115.760915 5637.0 5752.25 5834.0 5934.50 6058.0
    honggfuzz 82800 20.0 5717.000000 79.911793 5587.0 5642.00 5761.5 5783.50 5823.0
    aflplusplus 82800 13.0 5611.615385 226.340650 5156.0 5467.00 5613.0 5804.00 5922.0
    libfuzzer 82800 20.0 5542.450000 131.661238 5295.0 5460.75 5512.0 5626.25 5809.0
    fairfuzz 82800 20.0 4977.950000 123.945436 4791.0 4903.50 4990.0 5019.75 5351.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

harfbuzz_hb-shape-fuzzer_17863b summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 1.00 0.000000 1.0 1.0 1.0 1.0 1.0
    aflfast 82800 20.0 1.00 0.000000 1.0 1.0 1.0 1.0 1.0
    aflplusplus 82800 20.0 1.00 0.000000 1.0 1.0 1.0 1.0 1.0
    aflsmart 82800 20.0 1.00 0.000000 1.0 1.0 1.0 1.0 1.0
    centipede 82800 17.0 1.00 0.000000 1.0 1.0 1.0 1.0 1.0
    eclipser 82800 20.0 1.00 0.000000 1.0 1.0 1.0 1.0 1.0
    fairfuzz 82800 20.0 0.95 0.223607 0.0 1.0 1.0 1.0 1.0
    honggfuzz 82800 20.0 1.00 0.000000 1.0 1.0 1.0 1.0 1.0
    libafl 82800 20.0 1.00 0.000000 1.0 1.0 1.0 1.0 1.0
    libfuzzer 82800 20.0 1.00 0.000000 1.0 1.0 1.0 1.0 1.0
    mopt 82800 20.0 1.00 0.000000 1.0 1.0 1.0 1.0 1.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libafl 82800 20.0 10442.300000 27.836175 10409.0 10421.00 10440.0 10449.00 10505.0
    honggfuzz 82800 20.0 10151.400000 51.681616 10062.0 10112.25 10147.5 10196.00 10233.0
    mopt 82800 20.0 10143.100000 42.985799 10029.0 10122.00 10144.5 10175.25 10197.0
    eclipser 82800 20.0 10141.100000 52.770506 10030.0 10114.50 10141.0 10171.00 10216.0
    aflsmart 82800 20.0 10118.100000 33.329140 10037.0 10101.75 10120.0 10142.25 10166.0
    afl 82800 20.0 10112.750000 52.843239 10008.0 10085.25 10118.0 10135.50 10219.0
    aflfast 82800 20.0 10071.450000 52.709207 9953.0 10038.50 10063.0 10112.50 10172.0
    aflplusplus 82800 20.0 10039.150000 41.948935 9958.0 10010.75 10036.0 10061.50 10115.0
    libfuzzer 82800 20.0 9898.300000 49.400724 9786.0 9870.00 9891.5 9929.00 9985.0
    fairfuzz 82800 20.0 9285.200000 407.084060 8331.0 9086.00 9398.5 9589.50 9837.0
    centipede 82800 17.0 9198.764706 137.476148 9024.0 9079.00 9167.0 9284.00 9464.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

libxml2_xml_e85b9b summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 0.950000 0.223607 0.0 1.0 1.0 1.00 1.0
    aflfast 82800 20.0 0.950000 0.223607 0.0 1.0 1.0 1.00 1.0
    aflsmart 82800 20.0 0.950000 0.223607 0.0 1.0 1.0 1.00 1.0
    eclipser 82800 20.0 1.000000 0.000000 1.0 1.0 1.0 1.00 1.0
    fairfuzz 82800 20.0 0.700000 0.470162 0.0 0.0 1.0 1.00 1.0
    libafl 82800 19.0 1.000000 0.000000 1.0 1.0 1.0 1.00 1.0
    mopt 82800 20.0 1.000000 0.000000 1.0 1.0 1.0 1.00 1.0
    aflplusplus 82800 20.0 0.250000 0.444262 0.0 0.0 0.0 0.25 1.0
    centipede 82800 19.0 0.421053 0.507257 0.0 0.0 0.0 1.00 1.0
    honggfuzz 82800 20.0 0.450000 0.510418 0.0 0.0 0.0 1.00 1.0
    libfuzzer 82800 20.0 0.400000 0.502625 0.0 0.0 0.0 1.00 1.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libafl 82800 19.0 19895.210526 84.521252 19698.0 19847.00 19926.0 19960.00 19994.0
    eclipser 82800 20.0 19546.500000 661.319693 16858.0 19570.50 19685.5 19813.25 20007.0
    afl 82800 20.0 19662.850000 157.257508 19344.0 19565.00 19673.0 19745.50 19976.0
    mopt 82800 20.0 19616.550000 141.313938 19390.0 19500.00 19653.0 19738.25 19844.0
    aflsmart 82800 20.0 19634.100000 165.270400 19241.0 19559.00 19627.0 19700.00 19949.0
    aflfast 82800 20.0 19386.800000 631.283204 16851.0 19382.50 19528.0 19659.50 19903.0
    fairfuzz 82800 20.0 18810.600000 1337.762130 15320.0 19091.00 19311.0 19614.75 19790.0
    aflplusplus 82800 20.0 19083.150000 628.364064 16614.0 19091.50 19251.0 19308.50 19663.0
    honggfuzz 82800 20.0 17066.700000 76.783427 16941.0 17018.75 17052.0 17125.00 17239.0
    libfuzzer 82800 20.0 17068.850000 1077.021002 16146.0 16308.75 16446.5 18156.00 19077.0
    centipede 82800 19.0 16371.842105 780.048878 15491.0 15806.00 16334.0 16488.00 18398.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

mbedtls_fuzz_dtlsclient_7c6b0e summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    aflfast 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    aflplusplus 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    aflsmart 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    centipede 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    eclipser 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    fairfuzz 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    honggfuzz 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    libafl 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    libfuzzer 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    mopt 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libafl 82800 20.0 3003.25 363.432738 2555.0 2585.75 3033.5 3363.25 3658.0
    aflplusplus 82800 20.0 2615.35 11.389168 2587.0 2612.00 2617.0 2621.25 2631.0
    fairfuzz 82800 20.0 2693.20 309.228684 2477.0 2589.25 2614.5 2624.50 3599.0
    afl 82800 20.0 2578.10 66.260967 2419.0 2587.50 2605.0 2610.50 2624.0
    eclipser 82800 20.0 2583.50 56.592263 2413.0 2580.00 2599.5 2612.25 2639.0
    mopt 82800 20.0 2548.30 96.503668 2380.0 2448.50 2599.0 2614.00 2641.0
    aflsmart 82800 20.0 2586.75 39.238642 2440.0 2582.50 2593.5 2602.50 2629.0
    honggfuzz 82800 20.0 2618.90 126.658638 2557.0 2577.00 2589.5 2605.50 3151.0
    libfuzzer 82800 20.0 2576.50 14.536705 2550.0 2566.50 2575.0 2587.00 2612.0
    centipede 82800 20.0 2534.50 18.821880 2485.0 2524.00 2533.0 2543.25 2577.0
    aflfast 82800 20.0 2504.70 78.712067 2257.0 2509.50 2530.0 2545.25 2559.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

php_php-fuzz-parser_0dbedb summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    centipede 82800 20.0 1.05 0.223607 1.0 1.0 1.0 1.00 2.0
    afl 82800 20.0 0.00 0.000000 0.0 0.0 0.0 0.00 0.0
    aflfast 82800 20.0 0.00 0.000000 0.0 0.0 0.0 0.00 0.0
    aflplusplus 82800 20.0 0.25 0.444262 0.0 0.0 0.0 0.25 1.0
    aflsmart 82800 20.0 0.20 0.410391 0.0 0.0 0.0 0.00 1.0
    eclipser 82800 20.0 0.05 0.223607 0.0 0.0 0.0 0.00 1.0
    fairfuzz 82800 20.0 0.00 0.000000 0.0 0.0 0.0 0.00 0.0
    honggfuzz 82800 20.0 0.10 0.307794 0.0 0.0 0.0 0.00 1.0
    libafl 82800 20.0 0.05 0.223607 0.0 0.0 0.0 0.00 1.0
    libfuzzer 82800 20.0 0.10 0.307794 0.0 0.0 0.0 0.00 1.0
    mopt 82800 20.0 0.05 0.223607 0.0 0.0 0.0 0.00 1.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    centipede 82800 20.0 17698.45 244.461056 17421.0 17520.75 17595.5 17843.00 18188.0
    honggfuzz 82800 20.0 17061.25 77.155943 16875.0 17027.50 17079.5 17111.50 17170.0
    aflplusplus 82800 20.0 16908.50 137.399724 16755.0 16788.00 16865.0 17055.50 17132.0
    libafl 82800 20.0 16755.60 26.242593 16708.0 16748.00 16756.0 16765.75 16808.0
    libfuzzer 82800 20.0 16750.05 43.326757 16686.0 16718.75 16747.0 16768.00 16851.0
    aflsmart 82800 20.0 16583.95 67.721158 16445.0 16544.00 16612.0 16632.25 16701.0
    eclipser 82800 20.0 16612.90 42.046215 16537.0 16595.75 16608.5 16633.50 16727.0
    afl 82800 20.0 16603.85 30.783240 16550.0 16585.00 16606.0 16621.50 16674.0
    mopt 82800 20.0 16556.75 64.559132 16384.0 16546.50 16560.0 16592.50 16659.0
    aflfast 82800 20.0 16541.40 46.162642 16447.0 16505.50 16557.0 16571.25 16614.0
    fairfuzz 82800 20.0 16290.15 247.908785 15724.0 16146.75 16286.0 16407.00 16857.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

experiment data

You can download the raw data for this report here.

Check out the documentation on how to create customized reports using this data. Also see some example Colab notebooks for doing custom analysis on the data here.

Experiment Description:

(None,)