FuzzBench: 2023-12-15-fishfuzz-bug report

experiment summary

We show two different aggregate (cross-benchmark) rankings of fuzzers. The first is based on the average of per-benchmarks scores, where the score represents the percentage of the highest reached median bug-coverage on a given benchmark (higher value is better). The second ranking shows the average rank of fuzzers, after we rank them on each benchmark according to their median reached bug-covereges (lower value is better).
By avg. score
average normalized score
fuzzer
afl 75.0
aflfast 75.0
aflsmart 75.0
eclipser 75.0
fishpp_new_fast 75.0
libafl 75.0
mopt 75.0
aflplusplus 50.0
centipede 50.0
fairfuzz 50.0
fishpp_new 50.0
fishpp_new_exp 50.0
libfuzzer 50.0
honggfuzz 25.0
By avg. rank
average rank
fuzzer
afl 1.2
aflfast 1.2
aflsmart 1.2
eclipser 1.2
fishpp_new_fast 1.2
libafl 1.2
mopt 1.2
aflplusplus 2.8
fishpp_new 2.8
fishpp_new_exp 2.8
libfuzzer 2.8
fairfuzz 3.4
centipede 4.8
honggfuzz 5.0
  • Critical difference diagram
    The diagram visualizes the average rank of fuzzers (second ranking above) while showing the significance of the differences as well. What is considered a "critical difference" (CD) is based on the Friedman/Nemenyi post-hoc test. See more in the documentation.
    Note: If a fuzzer does not support all benchmarks, its ranking as shown in this diagram can be lower than it should be. So please check the list of supported benchmarks for the fuzzer(s) of your interest. The list could be specified in the fuzzer's README.md like this.
  • Median relative code-coverages on each benchmark

    Note: The relative coverage summary table shows the median relative performance of each fuzzer to the experiment maximum. Thus the highest relative performance may not be 100%.
    trial_relative_coverage = trial_coverage / experiment_max_coverage

      libafl mopt afl eclipser aflsmart aflfast fishpp_new_exp fishpp_new_fast aflplusplus fishpp_new honggfuzz libfuzzer fairfuzz centipede
    FuzzerMedian 95.00 96.00 95.00 94.00 94.00 93.00 92.00 92.00 92.00 92.00 92.00 88.00 89.00 84.00
    FuzzerMean 93.40 90.40 90.20 90.00 89.80 89.00 89.00 89.00 88.60 88.60 87.20 85.20 85.00 83.25
    bloaty_fuzz_target_52948c 95.00 96.00 95.00 94.00 94.00 93.00 91.00 91.00 90.00 89.00 92.00 88.00 80.00 nan
    harfbuzz_hb-shape-fuzzer_17863b 99.00 96.00 96.00 96.00 96.00 95.00 95.00 95.00 95.00 95.00 96.00 94.00 89.00 87.00
    libxml2_xml_e85b9b 99.00 98.00 98.00 98.00 98.00 97.00 96.00 96.00 95.00 96.00 85.00 82.00 96.00 81.00
    mbedtls_fuzz_dtlsclient_7c6b0e 82.00 71.00 71.00 71.00 70.00 69.00 71.00 71.00 71.00 71.00 70.00 70.00 71.00 69.00
    php_php-fuzz-parser_0dbedb 92.00 91.00 91.00 91.00 91.00 91.00 92.00 92.00 92.00 92.00 93.00 92.00 89.00 96.00
    • Fuzzers are sorted by "FuzzerMean" (average median relative coverage), highest on the left.
    • Green background = highest relative median coverage.
    • Blue gradient background = greater than 95% relative median coverage.
  • Median relative bug-coverages on each benchmark

    Note: The relative coverage summary table shows the median relative performance of each fuzzer to the experiment maximum. Thus the highest relative performance may not be 100%.
    trial_relative_coverage = trial_coverage / experiment_max_coverage

      afl aflfast aflsmart eclipser fishpp_new_fast libafl mopt aflplusplus fairfuzz fishpp_new fishpp_new_exp libfuzzer centipede honggfuzz
    FuzzerMedian 100.00 100.00 100.00 100.00 100.00 100.00 100.00 0.00 0.00 0.00 0.00 0.00 25.00 0.00
    FuzzerMean 60.00 60.00 60.00 60.00 60.00 60.00 60.00 40.00 40.00 40.00 40.00 40.00 37.50 20.00
    bloaty_fuzz_target_52948c 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 0.00 100.00 100.00 100.00 nan 0.00
    harfbuzz_hb-shape-fuzzer_17863b 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00
    libxml2_xml_e85b9b 100.00 100.00 100.00 100.00 100.00 100.00 100.00 0.00 100.00 0.00 0.00 0.00 0.00 0.00
    mbedtls_fuzz_dtlsclient_7c6b0e 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
    php_php-fuzz-parser_0dbedb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 50.00 0.00
    • Fuzzers are sorted by "FuzzerMean" (average median relative coverage), highest on the left.
    • Green background = highest relative median coverage.
    • Blue gradient background = greater than 95% relative median coverage.
  • Total unique bugs found on each benchmark
      Total libafl libfuzzer afl aflsmart eclipser fishpp_new_exp mopt aflfast centipede aflplusplus fairfuzz fishpp_new fishpp_new_fast honggfuzz
    FuzzerSum 13 10 10 9 9 9 9 9 8 8 7 7 7 7 7
    bloaty_fuzz_target_52948c 1 1 1 1 1 1 1 1 1 0 1 1 1 1 1
    harfbuzz_hb-shape-fuzzer_17863b 7 6 6 6 5 4 5 5 5 3 4 4 3 3 3
    libxml2_xml_e85b9b 2 2 2 2 2 2 2 2 2 2 1 2 2 2 2
    mbedtls_fuzz_dtlsclient_7c6b0e 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
    php_php-fuzz-parser_0dbedb 3 1 1 0 1 2 1 1 0 3 1 0 1 1 1
    • Fuzzers are sorted by "FuzzerSum", highest on the left.
    • Green background = most unique bugs found.
    • *note: This table represents unique bugs found across all trials.

bloaty_fuzz_target_52948c summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
error
The following fuzzers do not have enough samples: aflplusplus, fishpp_new_fast, fishpp_new_exp, fishpp_new.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 1.000000 0.000000 1.0 1.0 1.0 1.0 1.0
    aflfast 82800 20.0 0.950000 0.223607 0.0 1.0 1.0 1.0 1.0
    aflplusplus 82800 15.0 0.933333 0.258199 0.0 1.0 1.0 1.0 1.0
    aflsmart 82800 20.0 1.000000 0.000000 1.0 1.0 1.0 1.0 1.0
    eclipser 82800 20.0 1.000000 0.000000 1.0 1.0 1.0 1.0 1.0
    fishpp_new 82800 11.0 1.000000 0.000000 1.0 1.0 1.0 1.0 1.0
    fishpp_new_exp 82800 12.0 0.916667 0.288675 0.0 1.0 1.0 1.0 1.0
    fishpp_new_fast 82800 14.0 0.928571 0.267261 0.0 1.0 1.0 1.0 1.0
    libafl 82800 20.0 0.900000 0.307794 0.0 1.0 1.0 1.0 1.0
    libfuzzer 82800 20.0 0.900000 0.307794 0.0 1.0 1.0 1.0 1.0
    mopt 82800 20.0 1.000000 0.000000 1.0 1.0 1.0 1.0 1.0
    fairfuzz 82800 20.0 0.050000 0.223607 0.0 0.0 0.0 0.0 1.0
    honggfuzz 82800 20.0 0.350000 0.489360 0.0 0.0 0.0 1.0 1.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    mopt 82800 20.0 6011.800000 113.852906 5855.0 5912.50 6019.0 6091.50 6235.0
    afl 82800 20.0 5888.550000 114.767717 5690.0 5794.00 5937.0 5977.50 6069.0
    libafl 82800 20.0 5703.650000 469.483285 4372.0 5485.00 5936.0 6046.75 6140.0
    aflsmart 82800 20.0 5911.050000 79.416606 5766.0 5856.50 5918.5 5974.75 6033.0
    eclipser 82800 20.0 5867.800000 114.166822 5655.0 5769.25 5901.0 5963.50 6002.0
    aflfast 82800 20.0 5833.200000 115.760915 5637.0 5752.25 5834.0 5934.50 6058.0
    honggfuzz 82800 20.0 5717.000000 79.911793 5587.0 5642.00 5761.5 5783.50 5823.0
    fishpp_new_fast 82800 14.0 5731.000000 123.955327 5552.0 5638.25 5728.5 5838.25 5938.0
    fishpp_new_exp 82800 12.0 5681.166667 142.181595 5463.0 5587.75 5679.5 5772.50 5897.0
    aflplusplus 82800 15.0 5662.533333 164.831025 5407.0 5528.00 5612.0 5788.50 5957.0
    fishpp_new 82800 11.0 5634.181818 171.921388 5395.0 5559.00 5581.0 5701.00 5956.0
    libfuzzer 82800 20.0 5542.450000 131.661238 5295.0 5460.75 5512.0 5626.25 5809.0
    fairfuzz 82800 20.0 4977.950000 123.945436 4791.0 4903.50 4990.0 5019.75 5351.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

harfbuzz_hb-shape-fuzzer_17863b summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 1.00 0.000000 1.0 1.0 1.0 1.0 1.0
    aflfast 82800 20.0 1.00 0.000000 1.0 1.0 1.0 1.0 1.0
    aflplusplus 82800 20.0 0.95 0.223607 0.0 1.0 1.0 1.0 1.0
    aflsmart 82800 20.0 1.00 0.000000 1.0 1.0 1.0 1.0 1.0
    centipede 82800 17.0 1.00 0.000000 1.0 1.0 1.0 1.0 1.0
    eclipser 82800 20.0 1.00 0.000000 1.0 1.0 1.0 1.0 1.0
    fairfuzz 82800 20.0 0.95 0.223607 0.0 1.0 1.0 1.0 1.0
    fishpp_new 82800 20.0 1.00 0.000000 1.0 1.0 1.0 1.0 1.0
    fishpp_new_exp 82800 20.0 1.00 0.000000 1.0 1.0 1.0 1.0 1.0
    fishpp_new_fast 82800 20.0 1.00 0.000000 1.0 1.0 1.0 1.0 1.0
    honggfuzz 82800 20.0 1.00 0.000000 1.0 1.0 1.0 1.0 1.0
    libafl 82800 20.0 1.00 0.000000 1.0 1.0 1.0 1.0 1.0
    libfuzzer 82800 20.0 1.00 0.000000 1.0 1.0 1.0 1.0 1.0
    mopt 82800 20.0 1.00 0.000000 1.0 1.0 1.0 1.0 1.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libafl 82800 20.0 10442.300000 27.836175 10409.0 10421.00 10440.0 10449.00 10505.0
    honggfuzz 82800 20.0 10151.400000 51.681616 10062.0 10112.25 10147.5 10196.00 10233.0
    mopt 82800 20.0 10143.100000 42.985799 10029.0 10122.00 10144.5 10175.25 10197.0
    eclipser 82800 20.0 10141.100000 52.770506 10030.0 10114.50 10141.0 10171.00 10216.0
    aflsmart 82800 20.0 10118.100000 33.329140 10037.0 10101.75 10120.0 10142.25 10166.0
    afl 82800 20.0 10112.750000 52.843239 10008.0 10085.25 10118.0 10135.50 10219.0
    fishpp_new_exp 82800 20.0 10050.550000 141.550986 9472.0 10052.00 10079.0 10105.25 10166.0
    aflplusplus 82800 20.0 9945.050000 505.418167 7809.0 10004.00 10071.5 10092.00 10145.0
    aflfast 82800 20.0 10071.450000 52.709207 9953.0 10038.50 10063.0 10112.50 10172.0
    fishpp_new 82800 20.0 10018.500000 127.005802 9596.0 9981.25 10056.5 10088.00 10178.0
    fishpp_new_fast 82800 20.0 9998.500000 162.549588 9420.0 9962.75 10039.5 10090.25 10177.0
    libfuzzer 82800 20.0 9898.300000 49.400724 9786.0 9870.00 9891.5 9929.00 9985.0
    fairfuzz 82800 20.0 9285.200000 407.084060 8331.0 9086.00 9398.5 9589.50 9837.0
    centipede 82800 17.0 9198.764706 137.476148 9024.0 9079.00 9167.0 9284.00 9464.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

libxml2_xml_e85b9b summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 0.950000 0.223607 0.0 1.0 1.0 1.0 1.0
    aflfast 82800 20.0 0.950000 0.223607 0.0 1.0 1.0 1.0 1.0
    aflsmart 82800 20.0 0.950000 0.223607 0.0 1.0 1.0 1.0 1.0
    eclipser 82800 20.0 1.000000 0.000000 1.0 1.0 1.0 1.0 1.0
    fairfuzz 82800 20.0 0.700000 0.470162 0.0 0.0 1.0 1.0 1.0
    fishpp_new_fast 82800 20.0 0.550000 0.510418 0.0 0.0 1.0 1.0 1.0
    libafl 82800 19.0 1.000000 0.000000 1.0 1.0 1.0 1.0 1.0
    mopt 82800 20.0 1.000000 0.000000 1.0 1.0 1.0 1.0 1.0
    aflplusplus 82800 20.0 0.200000 0.410391 0.0 0.0 0.0 0.0 1.0
    centipede 82800 19.0 0.421053 0.507257 0.0 0.0 0.0 1.0 1.0
    fishpp_new 82800 20.0 0.300000 0.470162 0.0 0.0 0.0 1.0 1.0
    fishpp_new_exp 82800 18.0 0.388889 0.501631 0.0 0.0 0.0 1.0 1.0
    honggfuzz 82800 20.0 0.450000 0.510418 0.0 0.0 0.0 1.0 1.0
    libfuzzer 82800 20.0 0.400000 0.502625 0.0 0.0 0.0 1.0 1.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libafl 82800 19.0 19895.210526 84.521252 19698.0 19847.00 19926.0 19960.00 19994.0
    eclipser 82800 20.0 19546.500000 661.319693 16858.0 19570.50 19685.5 19813.25 20007.0
    afl 82800 20.0 19662.850000 157.257508 19344.0 19565.00 19673.0 19745.50 19976.0
    mopt 82800 20.0 19616.550000 141.313938 19390.0 19500.00 19653.0 19738.25 19844.0
    aflsmart 82800 20.0 19634.100000 165.270400 19241.0 19559.00 19627.0 19700.00 19949.0
    aflfast 82800 20.0 19386.800000 631.283204 16851.0 19382.50 19528.0 19659.50 19903.0
    fairfuzz 82800 20.0 18810.600000 1337.762130 15320.0 19091.00 19311.0 19614.75 19790.0
    fishpp_new 82800 20.0 19293.550000 296.594083 18642.0 19220.75 19285.5 19474.75 19811.0
    fishpp_new_fast 82800 20.0 19287.200000 270.604508 18860.0 19144.25 19260.5 19387.00 19858.0
    fishpp_new_exp 82800 18.0 19237.111111 254.864414 18412.0 19177.50 19242.5 19404.00 19531.0
    aflplusplus 82800 20.0 19031.300000 744.352852 15986.0 19035.00 19172.0 19336.00 19587.0
    honggfuzz 82800 20.0 17066.700000 76.783427 16941.0 17018.75 17052.0 17125.00 17239.0
    libfuzzer 82800 20.0 17068.850000 1077.021002 16146.0 16308.75 16446.5 18156.00 19077.0
    centipede 82800 19.0 16371.842105 780.048878 15491.0 15806.00 16334.0 16488.00 18398.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

mbedtls_fuzz_dtlsclient_7c6b0e summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    aflfast 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    aflplusplus 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    aflsmart 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    centipede 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    eclipser 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    fairfuzz 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    fishpp_new 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    fishpp_new_exp 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    fishpp_new_fast 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    honggfuzz 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    libafl 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    libfuzzer 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    mopt 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libafl 82800 20.0 3003.25 363.432738 2555.0 2585.75 3033.5 3363.25 3658.0
    fishpp_new 82800 20.0 2628.45 15.122221 2605.0 2614.75 2629.5 2640.25 2653.0
    fishpp_new_fast 82800 20.0 2592.40 78.802485 2373.0 2594.75 2620.0 2636.00 2659.0
    fishpp_new_exp 82800 20.0 2594.90 68.015401 2406.0 2604.00 2619.0 2625.50 2652.0
    aflplusplus 82800 20.0 2617.50 13.678643 2586.0 2612.25 2618.5 2623.00 2643.0
    fairfuzz 82800 20.0 2693.20 309.228684 2477.0 2589.25 2614.5 2624.50 3599.0
    afl 82800 20.0 2578.10 66.260967 2419.0 2587.50 2605.0 2610.50 2624.0
    eclipser 82800 20.0 2583.50 56.592263 2413.0 2580.00 2599.5 2612.25 2639.0
    mopt 82800 20.0 2548.30 96.503668 2380.0 2448.50 2599.0 2614.00 2641.0
    aflsmart 82800 20.0 2586.75 39.238642 2440.0 2582.50 2593.5 2602.50 2629.0
    honggfuzz 82800 20.0 2618.90 126.658638 2557.0 2577.00 2589.5 2605.50 3151.0
    libfuzzer 82800 20.0 2576.50 14.536705 2550.0 2566.50 2575.0 2587.00 2612.0
    centipede 82800 20.0 2534.50 18.821880 2485.0 2524.00 2533.0 2543.25 2577.0
    aflfast 82800 20.0 2504.70 78.712067 2257.0 2509.50 2530.0 2545.25 2559.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

php_php-fuzz-parser_0dbedb summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    centipede 82800 20.0 1.05 0.223607 1.0 1.0 1.0 1.0 2.0
    afl 82800 20.0 0.00 0.000000 0.0 0.0 0.0 0.0 0.0
    aflfast 82800 20.0 0.00 0.000000 0.0 0.0 0.0 0.0 0.0
    aflplusplus 82800 20.0 0.15 0.366348 0.0 0.0 0.0 0.0 1.0
    aflsmart 82800 20.0 0.20 0.410391 0.0 0.0 0.0 0.0 1.0
    eclipser 82800 20.0 0.05 0.223607 0.0 0.0 0.0 0.0 1.0
    fairfuzz 82800 20.0 0.00 0.000000 0.0 0.0 0.0 0.0 0.0
    fishpp_new 82800 20.0 0.20 0.410391 0.0 0.0 0.0 0.0 1.0
    fishpp_new_exp 82800 20.0 0.30 0.470162 0.0 0.0 0.0 1.0 1.0
    fishpp_new_fast 82800 20.0 0.35 0.489360 0.0 0.0 0.0 1.0 1.0
    honggfuzz 82800 20.0 0.10 0.307794 0.0 0.0 0.0 0.0 1.0
    libafl 82800 20.0 0.05 0.223607 0.0 0.0 0.0 0.0 1.0
    libfuzzer 82800 20.0 0.10 0.307794 0.0 0.0 0.0 0.0 1.0
    mopt 82800 20.0 0.05 0.223607 0.0 0.0 0.0 0.0 1.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    centipede 82800 20.0 17698.45 244.461056 17421.0 17520.75 17595.5 17843.00 18188.0
    honggfuzz 82800 20.0 17061.25 77.155943 16875.0 17027.50 17079.5 17111.50 17170.0
    fishpp_new_fast 82800 20.0 16949.75 159.324520 16752.0 16823.75 16899.5 17098.00 17290.0
    aflplusplus 82800 20.0 16933.90 144.895570 16751.0 16816.25 16898.5 17047.25 17223.0
    fishpp_new 82800 20.0 16926.55 171.127061 16768.0 16795.75 16818.0 17085.50 17343.0
    fishpp_new_exp 82800 20.0 16830.85 100.671051 16746.0 16766.25 16791.0 16832.75 17051.0
    libafl 82800 20.0 16755.60 26.242593 16708.0 16748.00 16756.0 16765.75 16808.0
    libfuzzer 82800 20.0 16750.05 43.326757 16686.0 16718.75 16747.0 16768.00 16851.0
    aflsmart 82800 20.0 16583.95 67.721158 16445.0 16544.00 16612.0 16632.25 16701.0
    eclipser 82800 20.0 16612.90 42.046215 16537.0 16595.75 16608.5 16633.50 16727.0
    afl 82800 20.0 16603.85 30.783240 16550.0 16585.00 16606.0 16621.50 16674.0
    mopt 82800 20.0 16556.75 64.559132 16384.0 16546.50 16560.0 16592.50 16659.0
    aflfast 82800 20.0 16541.40 46.162642 16447.0 16505.50 16557.0 16571.25 16614.0
    fairfuzz 82800 20.0 16290.15 247.908785 15724.0 16146.75 16286.0 16407.00 16857.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

experiment data

You can download the raw data for this report here.

Check out the documentation on how to create customized reports using this data. Also see some example Colab notebooks for doing custom analysis on the data here.

Experiment Description:

(None,)