FuzzBench: 2023-12-11-fishfuzz-bug report

experiment summary

We show two different aggregate (cross-benchmark) rankings of fuzzers. The first is based on the average of per-benchmarks scores, where the score represents the percentage of the highest reached median bug-coverage on a given benchmark (higher value is better). The second ranking shows the average rank of fuzzers, after we rank them on each benchmark according to their median reached bug-covereges (lower value is better).
By avg. score
average normalized score
fuzzer
afl 75.0
aflfast 75.0
aflsmart 75.0
eclipser 75.0
libafl 75.0
mopt 75.0
aflplusplus 50.0
aflplusplus_nocmp 50.0
centipede 50.0
fairfuzz 50.0
fishpp_new_exp 50.0
fishpp_new_nocmp_exp 50.0
libfuzzer 50.0
honggfuzz 25.0
By avg. rank
average rank
fuzzer
afl 1.2
aflfast 1.2
aflsmart 1.2
eclipser 1.2
libafl 1.2
mopt 1.2
aflplusplus 2.6
aflplusplus_nocmp 2.6
fishpp_new_exp 2.6
fishpp_new_nocmp_exp 2.6
libfuzzer 2.6
fairfuzz 3.4
centipede 4.6
honggfuzz 4.8
  • Critical difference diagram
    The diagram visualizes the average rank of fuzzers (second ranking above) while showing the significance of the differences as well. What is considered a "critical difference" (CD) is based on the Friedman/Nemenyi post-hoc test. See more in the documentation.
    Note: If a fuzzer does not support all benchmarks, its ranking as shown in this diagram can be lower than it should be. So please check the list of supported benchmarks for the fuzzer(s) of your interest. The list could be specified in the fuzzer's README.md like this.
  • Median relative code-coverages on each benchmark

    Note: The relative coverage summary table shows the median relative performance of each fuzzer to the experiment maximum. Thus the highest relative performance may not be 100%.
    trial_relative_coverage = trial_coverage / experiment_max_coverage

      fishpp_new_exp libafl mopt afl eclipser aflsmart aflplusplus_nocmp aflfast fishpp_new_nocmp_exp aflplusplus honggfuzz libfuzzer fairfuzz centipede
    FuzzerMedian 93.50 95.00 96.00 95.00 94.00 94.00 92.00 93.00 92.00 93.00 92.00 88.00 89.00 84.00
    FuzzerMean 93.50 93.40 90.40 90.20 90.00 89.80 89.40 89.00 89.00 88.60 87.20 85.20 85.00 83.25
    bloaty_fuzz_target_52948c 91.00 95.00 96.00 95.00 94.00 94.00 92.00 93.00 91.00 89.00 92.00 88.00 80.00 nan
    harfbuzz_hb-shape-fuzzer_17863b 95.00 99.00 96.00 96.00 96.00 96.00 96.00 95.00 95.00 95.00 96.00 94.00 89.00 87.00
    libxml2_xml_e85b9b 96.00 99.00 98.00 98.00 98.00 98.00 96.00 97.00 96.00 95.00 85.00 82.00 96.00 81.00
    mbedtls_fuzz_dtlsclient_7c6b0e nan 82.00 71.00 71.00 71.00 70.00 71.00 69.00 71.00 71.00 70.00 70.00 71.00 69.00
    php_php-fuzz-parser_0dbedb 92.00 92.00 91.00 91.00 91.00 91.00 92.00 91.00 92.00 93.00 93.00 92.00 89.00 96.00
    • Fuzzers are sorted by "FuzzerMean" (average median relative coverage), highest on the left.
    • Green background = highest relative median coverage.
    • Blue gradient background = greater than 95% relative median coverage.
  • Median relative bug-coverages on each benchmark

    Note: The relative coverage summary table shows the median relative performance of each fuzzer to the experiment maximum. Thus the highest relative performance may not be 100%.
    trial_relative_coverage = trial_coverage / experiment_max_coverage

      afl aflfast aflsmart eclipser libafl mopt fishpp_new_nocmp_exp fishpp_new_exp aflplusplus aflplusplus_nocmp fairfuzz libfuzzer centipede honggfuzz
    FuzzerMedian 100.00 100.00 100.00 100.00 100.00 100.00 50.00 50.00 0.00 0.00 0.00 0.00 25.00 0.00
    FuzzerMean 60.00 60.00 60.00 60.00 60.00 60.00 55.00 50.00 40.00 40.00 40.00 40.00 37.50 20.00
    bloaty_fuzz_target_52948c 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 0.00 100.00 nan 0.00
    harfbuzz_hb-shape-fuzzer_17863b 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00 100.00
    libxml2_xml_e85b9b 100.00 100.00 100.00 100.00 100.00 100.00 50.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00
    mbedtls_fuzz_dtlsclient_7c6b0e 0.00 0.00 0.00 0.00 0.00 0.00 0.00 nan 0.00 0.00 0.00 0.00 0.00 0.00
    php_php-fuzz-parser_0dbedb 0.00 0.00 0.00 0.00 0.00 0.00 25.00 0.00 0.00 0.00 0.00 0.00 50.00 0.00
    • Fuzzers are sorted by "FuzzerMean" (average median relative coverage), highest on the left.
    • Green background = highest relative median coverage.
    • Blue gradient background = greater than 95% relative median coverage.
  • Total unique bugs found on each benchmark
      Total aflplusplus fishpp_new_exp libafl libfuzzer afl aflsmart eclipser mopt aflfast centipede fishpp_new_nocmp_exp aflplusplus_nocmp fairfuzz honggfuzz
    FuzzerSum 14 11 10 10 10 9 9 9 9 8 8 8 7 7 7
    bloaty_fuzz_target_52948c 1 1 1 1 1 1 1 1 1 1 0 1 1 1 1
    harfbuzz_hb-shape-fuzzer_17863b 7 5 5 6 6 6 5 4 5 5 3 4 4 4 3
    libxml2_xml_e85b9b 2 1 2 2 2 2 2 2 2 2 2 2 1 2 2
    mbedtls_fuzz_dtlsclient_7c6b0e 0 0 nan 0 0 0 0 0 0 0 0 0 0 0 0
    php_php-fuzz-parser_0dbedb 4 4 2 1 1 0 1 2 1 0 3 1 1 0 1
    • Fuzzers are sorted by "FuzzerSum", highest on the left.
    • Green background = most unique bugs found.
    • *note: This table represents unique bugs found across all trials.

bloaty_fuzz_target_52948c summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
error
The following fuzzers do not have enough samples: fishpp_new_nocmp_exp, aflplusplus, aflplusplus_nocmp, fishpp_new_exp.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 1.000000 0.000000 1.0 1.0 1.0 1.0 1.0
    aflfast 82800 20.0 0.950000 0.223607 0.0 1.0 1.0 1.0 1.0
    aflplusplus 82800 13.0 0.923077 0.277350 0.0 1.0 1.0 1.0 1.0
    aflplusplus_nocmp 82800 11.0 1.000000 0.000000 1.0 1.0 1.0 1.0 1.0
    aflsmart 82800 20.0 1.000000 0.000000 1.0 1.0 1.0 1.0 1.0
    eclipser 82800 20.0 1.000000 0.000000 1.0 1.0 1.0 1.0 1.0
    fishpp_new_exp 82800 10.0 1.000000 0.000000 1.0 1.0 1.0 1.0 1.0
    fishpp_new_nocmp_exp 82800 15.0 0.800000 0.414039 0.0 1.0 1.0 1.0 1.0
    libafl 82800 20.0 0.900000 0.307794 0.0 1.0 1.0 1.0 1.0
    libfuzzer 82800 20.0 0.900000 0.307794 0.0 1.0 1.0 1.0 1.0
    mopt 82800 20.0 1.000000 0.000000 1.0 1.0 1.0 1.0 1.0
    fairfuzz 82800 20.0 0.050000 0.223607 0.0 0.0 0.0 0.0 1.0
    honggfuzz 82800 20.0 0.350000 0.489360 0.0 0.0 0.0 1.0 1.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    mopt 82800 20.0 6011.800000 113.852906 5855.0 5912.50 6019.0 6091.50 6235.0
    afl 82800 20.0 5888.550000 114.767717 5690.0 5794.00 5937.0 5977.50 6069.0
    libafl 82800 20.0 5703.650000 469.483285 4372.0 5485.00 5936.0 6046.75 6140.0
    aflsmart 82800 20.0 5911.050000 79.416606 5766.0 5856.50 5918.5 5974.75 6033.0
    eclipser 82800 20.0 5867.800000 114.166822 5655.0 5769.25 5901.0 5963.50 6002.0
    aflfast 82800 20.0 5833.200000 115.760915 5637.0 5752.25 5834.0 5934.50 6058.0
    aflplusplus_nocmp 82800 11.0 5743.818182 171.474090 5444.0 5654.00 5768.0 5884.00 5947.0
    honggfuzz 82800 20.0 5717.000000 79.911793 5587.0 5642.00 5761.5 5783.50 5823.0
    fishpp_new_nocmp_exp 82800 15.0 5747.466667 99.548170 5599.0 5670.00 5721.0 5834.00 5920.0
    fishpp_new_exp 82800 10.0 5691.100000 125.803595 5483.0 5633.25 5718.5 5783.00 5833.0
    aflplusplus 82800 13.0 5593.230769 173.308566 5393.0 5443.00 5556.0 5718.00 5881.0
    libfuzzer 82800 20.0 5542.450000 131.661238 5295.0 5460.75 5512.0 5626.25 5809.0
    fairfuzz 82800 20.0 4977.950000 123.945436 4791.0 4903.50 4990.0 5019.75 5351.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

harfbuzz_hb-shape-fuzzer_17863b summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 1.000000 0.000000 1.0 1.0 1.0 1.0 1.0
    aflfast 82800 20.0 1.000000 0.000000 1.0 1.0 1.0 1.0 1.0
    aflplusplus 82800 20.0 1.000000 0.000000 1.0 1.0 1.0 1.0 1.0
    aflplusplus_nocmp 82800 19.0 0.947368 0.229416 0.0 1.0 1.0 1.0 1.0
    aflsmart 82800 20.0 1.000000 0.000000 1.0 1.0 1.0 1.0 1.0
    centipede 82800 17.0 1.000000 0.000000 1.0 1.0 1.0 1.0 1.0
    eclipser 82800 20.0 1.000000 0.000000 1.0 1.0 1.0 1.0 1.0
    fairfuzz 82800 20.0 0.950000 0.223607 0.0 1.0 1.0 1.0 1.0
    fishpp_new_exp 82800 20.0 1.000000 0.000000 1.0 1.0 1.0 1.0 1.0
    fishpp_new_nocmp_exp 82800 19.0 0.947368 0.229416 0.0 1.0 1.0 1.0 1.0
    honggfuzz 82800 20.0 1.000000 0.000000 1.0 1.0 1.0 1.0 1.0
    libafl 82800 20.0 1.000000 0.000000 1.0 1.0 1.0 1.0 1.0
    libfuzzer 82800 20.0 1.000000 0.000000 1.0 1.0 1.0 1.0 1.0
    mopt 82800 20.0 1.000000 0.000000 1.0 1.0 1.0 1.0 1.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libafl 82800 20.0 10442.300000 27.836175 10409.0 10421.00 10440.0 10449.00 10505.0
    honggfuzz 82800 20.0 10151.400000 51.681616 10062.0 10112.25 10147.5 10196.00 10233.0
    mopt 82800 20.0 10143.100000 42.985799 10029.0 10122.00 10144.5 10175.25 10197.0
    eclipser 82800 20.0 10141.100000 52.770506 10030.0 10114.50 10141.0 10171.00 10216.0
    aflsmart 82800 20.0 10118.100000 33.329140 10037.0 10101.75 10120.0 10142.25 10166.0
    afl 82800 20.0 10112.750000 52.843239 10008.0 10085.25 10118.0 10135.50 10219.0
    aflplusplus_nocmp 82800 19.0 9985.421053 360.069795 8521.0 10034.50 10089.0 10102.50 10148.0
    fishpp_new_exp 82800 20.0 10073.350000 70.048310 9904.0 10042.50 10078.0 10114.00 10200.0
    aflplusplus 82800 20.0 10074.100000 55.905842 9898.0 10053.50 10071.5 10114.00 10149.0
    aflfast 82800 20.0 10071.450000 52.709207 9953.0 10038.50 10063.0 10112.50 10172.0
    fishpp_new_nocmp_exp 82800 19.0 9954.052632 486.142809 7958.0 10016.50 10053.0 10094.00 10160.0
    libfuzzer 82800 20.0 9898.300000 49.400724 9786.0 9870.00 9891.5 9929.00 9985.0
    fairfuzz 82800 20.0 9285.200000 407.084060 8331.0 9086.00 9398.5 9589.50 9837.0
    centipede 82800 17.0 9198.764706 137.476148 9024.0 9079.00 9167.0 9284.00 9464.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

libxml2_xml_e85b9b summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 0.950000 0.223607 0.0 1.0 1.0 1.00 1.0
    aflfast 82800 20.0 0.950000 0.223607 0.0 1.0 1.0 1.00 1.0
    aflsmart 82800 20.0 0.950000 0.223607 0.0 1.0 1.0 1.00 1.0
    eclipser 82800 20.0 1.000000 0.000000 1.0 1.0 1.0 1.00 1.0
    fairfuzz 82800 20.0 0.700000 0.470162 0.0 0.0 1.0 1.00 1.0
    libafl 82800 19.0 1.000000 0.000000 1.0 1.0 1.0 1.00 1.0
    mopt 82800 20.0 1.000000 0.000000 1.0 1.0 1.0 1.00 1.0
    fishpp_new_nocmp_exp 82800 20.0 0.500000 0.512989 0.0 0.0 0.5 1.00 1.0
    aflplusplus 82800 19.0 0.052632 0.229416 0.0 0.0 0.0 0.00 1.0
    aflplusplus_nocmp 82800 20.0 0.250000 0.444262 0.0 0.0 0.0 0.25 1.0
    centipede 82800 19.0 0.421053 0.507257 0.0 0.0 0.0 1.00 1.0
    fishpp_new_exp 82800 18.0 0.388889 0.501631 0.0 0.0 0.0 1.00 1.0
    honggfuzz 82800 20.0 0.450000 0.510418 0.0 0.0 0.0 1.00 1.0
    libfuzzer 82800 20.0 0.400000 0.502625 0.0 0.0 0.0 1.00 1.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libafl 82800 19.0 19895.210526 84.521252 19698.0 19847.00 19926.0 19960.00 19994.0
    eclipser 82800 20.0 19546.500000 661.319693 16858.0 19570.50 19685.5 19813.25 20007.0
    afl 82800 20.0 19662.850000 157.257508 19344.0 19565.00 19673.0 19745.50 19976.0
    mopt 82800 20.0 19616.550000 141.313938 19390.0 19500.00 19653.0 19738.25 19844.0
    aflsmart 82800 20.0 19634.100000 165.270400 19241.0 19559.00 19627.0 19700.00 19949.0
    aflfast 82800 20.0 19386.800000 631.283204 16851.0 19382.50 19528.0 19659.50 19903.0
    fishpp_new_nocmp_exp 82800 20.0 19370.950000 281.904141 18605.0 19188.50 19327.5 19586.25 19869.0
    fairfuzz 82800 20.0 18810.600000 1337.762130 15320.0 19091.00 19311.0 19614.75 19790.0
    aflplusplus_nocmp 82800 20.0 19279.300000 228.385431 18490.0 19185.25 19306.0 19413.25 19602.0
    fishpp_new_exp 82800 18.0 19237.111111 254.864414 18412.0 19177.50 19242.5 19404.00 19531.0
    aflplusplus 82800 19.0 19114.315789 595.816531 16758.0 19105.50 19203.0 19354.50 19689.0
    honggfuzz 82800 20.0 17066.700000 76.783427 16941.0 17018.75 17052.0 17125.00 17239.0
    libfuzzer 82800 20.0 17068.850000 1077.021002 16146.0 16308.75 16446.5 18156.00 19077.0
    centipede 82800 19.0 16371.842105 780.048878 15491.0 15806.00 16334.0 16488.00 18398.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

mbedtls_fuzz_dtlsclient_7c6b0e summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    afl 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    aflfast 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    aflplusplus 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    aflplusplus_nocmp 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    aflsmart 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    centipede 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    eclipser 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    fairfuzz 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    fishpp_new_nocmp_exp 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    honggfuzz 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    libafl 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    libfuzzer 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
    mopt 82800 20.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    libafl 82800 20.0 3003.25 363.432738 2555.0 2585.75 3033.5 3363.25 3658.0
    fishpp_new_nocmp_exp 82800 20.0 2617.80 38.579515 2467.0 2612.75 2621.0 2635.50 2663.0
    aflplusplus 82800 20.0 2617.50 13.678643 2586.0 2612.25 2618.5 2623.00 2643.0
    fairfuzz 82800 20.0 2693.20 309.228684 2477.0 2589.25 2614.5 2624.50 3599.0
    afl 82800 20.0 2578.10 66.260967 2419.0 2587.50 2605.0 2610.50 2624.0
    eclipser 82800 20.0 2583.50 56.592263 2413.0 2580.00 2599.5 2612.25 2639.0
    mopt 82800 20.0 2548.30 96.503668 2380.0 2448.50 2599.0 2614.00 2641.0
    aflplusplus_nocmp 82800 20.0 2635.85 174.532541 2577.0 2589.50 2597.5 2605.75 3376.0
    aflsmart 82800 20.0 2586.75 39.238642 2440.0 2582.50 2593.5 2602.50 2629.0
    honggfuzz 82800 20.0 2618.90 126.658638 2557.0 2577.00 2589.5 2605.50 3151.0
    libfuzzer 82800 20.0 2576.50 14.536705 2550.0 2566.50 2575.0 2587.00 2612.0
    centipede 82800 20.0 2534.50 18.821880 2485.0 2524.00 2533.0 2543.25 2577.0
    aflfast 82800 20.0 2504.70 78.712067 2257.0 2509.50 2530.0 2545.25 2559.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

php_php-fuzz-parser_0dbedb summary

Discovered bug coverage distribution
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
Mean bug coverage growth over time
Mean bug coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (bugs covered)
    Bug coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    centipede 82800 20.0 1.05 0.223607 1.0 1.0 1.0 1.0 2.0
    fishpp_new_nocmp_exp 82800 20.0 0.50 0.512989 0.0 0.0 0.5 1.0 1.0
    afl 82800 20.0 0.00 0.000000 0.0 0.0 0.0 0.0 0.0
    aflfast 82800 20.0 0.00 0.000000 0.0 0.0 0.0 0.0 0.0
    aflplusplus 82800 20.0 0.55 0.825578 0.0 0.0 0.0 1.0 2.0
    aflplusplus_nocmp 82800 20.0 0.20 0.410391 0.0 0.0 0.0 0.0 1.0
    aflsmart 82800 20.0 0.20 0.410391 0.0 0.0 0.0 0.0 1.0
    eclipser 82800 20.0 0.05 0.223607 0.0 0.0 0.0 0.0 1.0
    fairfuzz 82800 20.0 0.00 0.000000 0.0 0.0 0.0 0.0 0.0
    fishpp_new_exp 82800 20.0 0.10 0.447214 0.0 0.0 0.0 0.0 2.0
    honggfuzz 82800 20.0 0.10 0.307794 0.0 0.0 0.0 0.0 1.0
    libafl 82800 20.0 0.05 0.223607 0.0 0.0 0.0 0.0 1.0
    libfuzzer 82800 20.0 0.10 0.307794 0.0 0.0 0.0 0.0 1.0
    mopt 82800 20.0 0.05 0.223607 0.0 0.0 0.0 0.0 1.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    centipede 82800 20.0 17698.45 244.461056 17421.0 17520.75 17595.5 17843.00 18188.0
    honggfuzz 82800 20.0 17061.25 77.155943 16875.0 17027.50 17079.5 17111.50 17170.0
    aflplusplus 82800 20.0 17041.10 185.677111 16773.0 16845.00 17061.0 17153.00 17373.0
    fishpp_new_nocmp_exp 82800 20.0 16895.60 119.677813 16736.0 16790.00 16909.0 16953.00 17236.0
    aflplusplus_nocmp 82800 20.0 16969.10 171.796360 16789.0 16828.75 16893.5 17091.00 17371.0
    fishpp_new_exp 82800 20.0 16871.55 127.684016 16748.0 16777.00 16806.5 17020.50 17118.0
    libafl 82800 20.0 16755.60 26.242593 16708.0 16748.00 16756.0 16765.75 16808.0
    libfuzzer 82800 20.0 16750.05 43.326757 16686.0 16718.75 16747.0 16768.00 16851.0
    aflsmart 82800 20.0 16583.95 67.721158 16445.0 16544.00 16612.0 16632.25 16701.0
    eclipser 82800 20.0 16612.90 42.046215 16537.0 16595.75 16608.5 16633.50 16727.0
    afl 82800 20.0 16603.85 30.783240 16550.0 16585.00 16606.0 16621.50 16674.0
    mopt 82800 20.0 16556.75 64.559132 16384.0 16546.50 16560.0 16592.50 16659.0
    aflfast 82800 20.0 16541.40 46.162642 16447.0 16505.50 16557.0 16571.25 16614.0
    fairfuzz 82800 20.0 16290.15 247.908785 15724.0 16146.75 16286.0 16407.00 16857.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.
  • Unique code coverage plots
    Ranking by unique code branches covered
    Each bar shows the total number of code branches found by a given fuzzer. The colored area shows the number of unique code branches (i.e., branches that were not covered by any other fuzzers).
    Pairwise unique code coverage
    Each cell represents the number of code branches covered by the fuzzer of the column but not by the fuzzer of the row

experiment data

You can download the raw data for this report here.

Check out the documentation on how to create customized reports using this data. Also see some example Colab notebooks for doing custom analysis on the data here.

Experiment Description:

(None,)