Class org.broadinstitute.hellbender.tools.spark.pipelines.MarkDuplicatesSparkIntegrationTest
|
100%
successful |
Tests
| Test | Duration | Result |
|---|---|---|
| assertThatExpectedOutputUpdateToggleIsDisabled | 0s | passed |
| testAssertAssumeUnsortedFilesAreQueryGroupedFiles | 0.548s | passed |
| testAssertCorrectSortOrderMultipleBams | 0.121s | passed |
| testAssertCorrectSortOrderMultipleBamsOverriding | 0.469s | passed |
| testBulkFragmentsNoDuplicates | 0.939s | passed |
| testBulkFragmentsWithDuplicates | 2.016s | passed |
| testDifferentChromosomesInOppositeOrder | 0.300s | passed |
| testDuplicateDetectionDataProviderWithMetrics[0](src/test/resources/org/broadinstitute/hellbender/tools/walkers/MarkDuplicatesGATK/example.chr1.1-1K.unmarkedDups.bam, src/test/resources/org/broadinstitute/hellbender/tools/walkers/MarkDuplicatesGATK/expected.picard-2.15.0.sorted.chr1.1-1K.unmarkedDups.markDuplicate.metrics) | 0.305s | passed |
| testDuplicateDetectionDataProviderWithMetrics[1](src/test/resources/org/broadinstitute/hellbender/tools/walkers/MarkDuplicatesGATK/optical_dupes.bam, src/test/resources/org/broadinstitute/hellbender/tools/walkers/MarkDuplicatesGATK/expected.picard-2.15.0.sorted.optical_dupes.markDuplicate.metrics) | 0.249s | passed |
| testDuplicateDetectionDataProviderWithMetrics[2](src/test/resources/org/broadinstitute/hellbender/tools/walkers/MarkDuplicatesGATK/inputSingleLibrarySolexa16404.bam, src/test/resources/org/broadinstitute/hellbender/tools/walkers/MarkDuplicatesGATK/expected.picard-2.15.0.sorted.inputSingleLibrarySolexa16404.metrics) | 0.257s | passed |
| testFlowModeFlag | 1.764s | passed |
| testHashCollisionHandling | 1.049s | passed |
| testMDOrder[0](src/test/resources/org/broadinstitute/hellbender/tools/walkers/MarkDuplicatesGATK/mdOrderBug.bam, src/test/resources/org/broadinstitute/hellbender/tools/walkers/MarkDuplicatesGATK/expected.mdOrderBug.bam) | 0.306s | passed |
| testMDOrder[1](src/test/resources/org/broadinstitute/hellbender/tools/walkers/MarkDuplicatesGATK/mdOrderBug2.bam, src/test/resources/org/broadinstitute/hellbender/tools/walkers/MarkDuplicatesGATK/expected.mdOrderBug2.bam) | 0.296s | passed |
| testManyOpticalDuplicateClusterOneEndSamePositionOneCluster[0](true, true, true) | 0.300s | passed |
| testManyOpticalDuplicateClusterOneEndSamePositionOneCluster[1](true, false, true) | 0.310s | passed |
| testManyOpticalDuplicateClusterOneEndSamePositionOneCluster[2](false, true, true) | 0.297s | passed |
| testManyOpticalDuplicateClusterOneEndSamePositionOneCluster[3](true, true, false) | 0.297s | passed |
| testManyOpticalDuplicateClusterOneEndSamePositionOneCluster[4](true, false, false) | 0.303s | passed |
| testManyOpticalDuplicateClusterOneEndSamePositionOneCluster[5](false, true, false) | 0.306s | passed |
| testMappedFragmentAndMappedPairFirstOfPairNonPrimary | 0.308s | passed |
| testMappedFragmentAndMatePairFirstUnmapped | 0.290s | passed |
| testMappedFragmentAndMatePairSecondUnmapped | 0.306s | passed |
| testMappedPairAndMappedFragmentAndMatePairFirstUnmapped | 0.292s | passed |
| testMappedPairAndMappedFragmentAndMatePairSecondUnmapped | 0.326s | passed |
| testMappedPairAndMatePairFirstOppositeStrandSecondUnmapped | 0.303s | passed |
| testMappedPairAndMatePairFirstUnmapped | 0.297s | passed |
| testMappedPairAndMatePairSecondUnmapped | 0.296s | passed |
| testMappedPairWithFirstEndSamePositionAndOther | 0.289s | passed |
| testMappedPairWithSamePosition | 0.301s | passed |
| testMappedPairWithSamePositionSameCigar | 0.288s | passed |
| testMarkDuplicatesSparkDeletingDuplicateReads[0]([Ljava.io.File;@32a1b035, 20, 0, {Solexa-16419=[0, 3, 0, 0, 0, 0, 0.0, 0], Solexa-16416=[0, 1, 0, 0, 0, 0, 0.0, 0], Solexa-16404=[0, 3, 0, 0, 0, 0, 0.0, 0], Solexa-16406=[0, 1, 0, 0, 0, 0, 0.0, 0], Solexa-16412=[0, 1, 0, 0, 0, 0, 0.0, 0]}) | 0.378s | passed |
| testMarkDuplicatesSparkDeletingDuplicateReads[1]([Ljava.io.File;@4b7aad9f, 90, 6, {Solexa-16419=[4, 4, 4, 0, 0, 0, 0.0, 0], Solexa-16416=[2, 2, 2, 0, 0, 0, 0.0, 0], Solexa-16404=[3, 9, 3, 0, 2, 0, 0.190476, 17], Solexa-16406=[1, 10, 1, 0, 0, 0, 0.0, 0], Solexa-16412=[3, 6, 3, 0, 1, 0, 0.133333, 15]}) | 0.406s | passed |
| testMarkDuplicatesSparkDeletingDuplicateReads[2]([Ljava.io.File;@2ed4a839, 90, 6, {Solexa-16419=[4, 4, 4, 0, 0, 0, 0.0, 0], Solexa-16416=[2, 2, 2, 0, 0, 0, 0.0, 0], Solexa-16404=[3, 9, 3, 0, 2, 0, 0.190476, 17], Solexa-16406=[1, 10, 1, 0, 0, 0, 0.0, 0], Solexa-16412=[3, 6, 3, 0, 1, 0, 0.133333, 15]}) | 0.364s | passed |
| testMarkDuplicatesSparkDeletingDuplicateReads[3]([Ljava.io.File;@6d1f64b8, 4, 2, {mylib=[0, 2, 0, 0, 1, 1, 0.5, 0]}) | 0.337s | passed |
| testMarkDuplicatesSparkDeletingDuplicateReads[4]([Ljava.io.File;@5002f21b, 4, 2, {mylib=[0, 2, 0, 0, 1, 1, 0.5, 0]}) | 0.325s | passed |
| testMarkDuplicatesSparkDeletingDuplicateReads[5]([Ljava.io.File;@41cc40ac, 94, 8, {mylib=[0, 2, 0, 0, 1, 1, 0.5, 0], Solexa-16419=[4, 4, 4, 0, 0, 0, 0.0, 0], Solexa-16416=[2, 2, 2, 0, 0, 0, 0.0, 0], Solexa-16404=[3, 9, 3, 0, 2, 0, 0.190476, 17], Solexa-16406=[1, 10, 1, 0, 0, 0, 0.0, 0], Solexa-16412=[3, 6, 3, 0, 1, 0, 0.133333, 15]}) | 1.191s | passed |
| testMarkDuplicatesSparkDeletingDuplicateReads[6]([Ljava.io.File;@3261d323, 94, 8, {mylib=[0, 2, 0, 0, 1, 1, 0.5, 0], Solexa-16419=[4, 4, 4, 0, 0, 0, 0.0, 0], Solexa-16416=[2, 2, 2, 0, 0, 0, 0.0, 0], Solexa-16404=[3, 9, 3, 0, 2, 0, 0.190476, 17], Solexa-16406=[1, 10, 1, 0, 0, 0, 0.0, 0], Solexa-16412=[3, 6, 3, 0, 1, 0, 0.133333, 15]}) | 1.162s | passed |
| testMarkDuplicatesSparkDeletingOpticalDuplicateReads[0]([Ljava.io.File;@25edf989, 20, 0, {Solexa-16419=[0, 3, 0, 0, 0, 0, 0.0, 0], Solexa-16416=[0, 1, 0, 0, 0, 0, 0.0, 0], Solexa-16404=[0, 3, 0, 0, 0, 0, 0.0, 0], Solexa-16406=[0, 1, 0, 0, 0, 0, 0.0, 0], Solexa-16412=[0, 1, 0, 0, 0, 0, 0.0, 0]}) | 0.370s | passed |
| testMarkDuplicatesSparkDeletingOpticalDuplicateReads[1]([Ljava.io.File;@1bb5ef33, 90, 6, {Solexa-16419=[4, 4, 4, 0, 0, 0, 0.0, 0], Solexa-16416=[2, 2, 2, 0, 0, 0, 0.0, 0], Solexa-16404=[3, 9, 3, 0, 2, 0, 0.190476, 17], Solexa-16406=[1, 10, 1, 0, 0, 0, 0.0, 0], Solexa-16412=[3, 6, 3, 0, 1, 0, 0.133333, 15]}) | 0.396s | passed |
| testMarkDuplicatesSparkDeletingOpticalDuplicateReads[2]([Ljava.io.File;@4bcd703c, 90, 6, {Solexa-16419=[4, 4, 4, 0, 0, 0, 0.0, 0], Solexa-16416=[2, 2, 2, 0, 0, 0, 0.0, 0], Solexa-16404=[3, 9, 3, 0, 2, 0, 0.190476, 17], Solexa-16406=[1, 10, 1, 0, 0, 0, 0.0, 0], Solexa-16412=[3, 6, 3, 0, 1, 0, 0.133333, 15]}) | 0.399s | passed |
| testMarkDuplicatesSparkDeletingOpticalDuplicateReads[3]([Ljava.io.File;@1f4f8fc0, 4, 2, {mylib=[0, 2, 0, 0, 1, 1, 0.5, 0]}) | 0.332s | passed |
| testMarkDuplicatesSparkDeletingOpticalDuplicateReads[4]([Ljava.io.File;@45ea798, 4, 2, {mylib=[0, 2, 0, 0, 1, 1, 0.5, 0]}) | 0.320s | passed |
| testMarkDuplicatesSparkDeletingOpticalDuplicateReads[5]([Ljava.io.File;@63ebf77e, 94, 8, {mylib=[0, 2, 0, 0, 1, 1, 0.5, 0], Solexa-16419=[4, 4, 4, 0, 0, 0, 0.0, 0], Solexa-16416=[2, 2, 2, 0, 0, 0, 0.0, 0], Solexa-16404=[3, 9, 3, 0, 2, 0, 0.190476, 17], Solexa-16406=[1, 10, 1, 0, 0, 0, 0.0, 0], Solexa-16412=[3, 6, 3, 0, 1, 0, 0.133333, 15]}) | 1.191s | passed |
| testMarkDuplicatesSparkDeletingOpticalDuplicateReads[6]([Ljava.io.File;@764a6e8b, 94, 8, {mylib=[0, 2, 0, 0, 1, 1, 0.5, 0], Solexa-16419=[4, 4, 4, 0, 0, 0, 0.0, 0], Solexa-16416=[2, 2, 2, 0, 0, 0, 0.0, 0], Solexa-16404=[3, 9, 3, 0, 2, 0, 0.190476, 17], Solexa-16406=[1, 10, 1, 0, 0, 0, 0.0, 0], Solexa-16412=[3, 6, 3, 0, 1, 0, 0.133333, 15]}) | 1.206s | passed |
| testMarkDuplicatesSparkIntegrationTestLocal[0]([Ljava.io.File;@3ea794cc, 20, 0, {Solexa-16419=[0, 3, 0, 0, 0, 0, 0.0, 0], Solexa-16416=[0, 1, 0, 0, 0, 0, 0.0, 0], Solexa-16404=[0, 3, 0, 0, 0, 0, 0.0, 0], Solexa-16406=[0, 1, 0, 0, 0, 0, 0.0, 0], Solexa-16412=[0, 1, 0, 0, 0, 0, 0.0, 0]}) | 0.337s | passed |
| testMarkDuplicatesSparkIntegrationTestLocal[1]([Ljava.io.File;@152925e5, 90, 6, {Solexa-16419=[4, 4, 4, 0, 0, 0, 0.0, 0], Solexa-16416=[2, 2, 2, 0, 0, 0, 0.0, 0], Solexa-16404=[3, 9, 3, 0, 2, 0, 0.190476, 17], Solexa-16406=[1, 10, 1, 0, 0, 0, 0.0, 0], Solexa-16412=[3, 6, 3, 0, 1, 0, 0.133333, 15]}) | 0.348s | passed |
| testMarkDuplicatesSparkIntegrationTestLocal[2]([Ljava.io.File;@6ea662, 90, 6, {Solexa-16419=[4, 4, 4, 0, 0, 0, 0.0, 0], Solexa-16416=[2, 2, 2, 0, 0, 0, 0.0, 0], Solexa-16404=[3, 9, 3, 0, 2, 0, 0.190476, 17], Solexa-16406=[1, 10, 1, 0, 0, 0, 0.0, 0], Solexa-16412=[3, 6, 3, 0, 1, 0, 0.133333, 15]}) | 0.349s | passed |
| testMarkDuplicatesSparkIntegrationTestLocal[3]([Ljava.io.File;@30b7eef1, 4, 2, {mylib=[0, 2, 0, 0, 1, 1, 0.5, 0]}) | 0.350s | passed |
| testMarkDuplicatesSparkIntegrationTestLocal[4]([Ljava.io.File;@137e0a14, 4, 2, {mylib=[0, 2, 0, 0, 1, 1, 0.5, 0]}) | 0.330s | passed |
| testMarkDuplicatesSparkIntegrationTestLocal[5]([Ljava.io.File;@38881182, 94, 8, {mylib=[0, 2, 0, 0, 1, 1, 0.5, 0], Solexa-16419=[4, 4, 4, 0, 0, 0, 0.0, 0], Solexa-16416=[2, 2, 2, 0, 0, 0, 0.0, 0], Solexa-16404=[3, 9, 3, 0, 2, 0, 0.190476, 17], Solexa-16406=[1, 10, 1, 0, 0, 0, 0.0, 0], Solexa-16412=[3, 6, 3, 0, 1, 0, 0.133333, 15]}) | 1.167s | passed |
| testMarkDuplicatesSparkIntegrationTestLocal[6]([Ljava.io.File;@63e7af4c, 94, 8, {mylib=[0, 2, 0, 0, 1, 1, 0.5, 0], Solexa-16419=[4, 4, 4, 0, 0, 0, 0.0, 0], Solexa-16416=[2, 2, 2, 0, 0, 0, 0.0, 0], Solexa-16404=[3, 9, 3, 0, 2, 0, 0.190476, 17], Solexa-16406=[1, 10, 1, 0, 0, 0, 0.0, 0], Solexa-16412=[3, 6, 3, 0, 1, 0, 0.133333, 15]}) | 1.164s | passed |
| testMarkDuplicatesSparkMarkingAllDuplicatesWithTagging[0]([Ljava.io.File;@2ecaae99, 20, 0, {Solexa-16419=[0, 3, 0, 0, 0, 0, 0.0, 0], Solexa-16416=[0, 1, 0, 0, 0, 0, 0.0, 0], Solexa-16404=[0, 3, 0, 0, 0, 0, 0.0, 0], Solexa-16406=[0, 1, 0, 0, 0, 0, 0.0, 0], Solexa-16412=[0, 1, 0, 0, 0, 0, 0.0, 0]}) | 0.338s | passed |
| testMarkDuplicatesSparkMarkingAllDuplicatesWithTagging[1]([Ljava.io.File;@4e00eb47, 90, 6, {Solexa-16419=[4, 4, 4, 0, 0, 0, 0.0, 0], Solexa-16416=[2, 2, 2, 0, 0, 0, 0.0, 0], Solexa-16404=[3, 9, 3, 0, 2, 0, 0.190476, 17], Solexa-16406=[1, 10, 1, 0, 0, 0, 0.0, 0], Solexa-16412=[3, 6, 3, 0, 1, 0, 0.133333, 15]}) | 0.347s | passed |
| testMarkDuplicatesSparkMarkingAllDuplicatesWithTagging[2]([Ljava.io.File;@7f109278, 90, 6, {Solexa-16419=[4, 4, 4, 0, 0, 0, 0.0, 0], Solexa-16416=[2, 2, 2, 0, 0, 0, 0.0, 0], Solexa-16404=[3, 9, 3, 0, 2, 0, 0.190476, 17], Solexa-16406=[1, 10, 1, 0, 0, 0, 0.0, 0], Solexa-16412=[3, 6, 3, 0, 1, 0, 0.133333, 15]}) | 0.346s | passed |
| testMarkDuplicatesSparkMarkingAllDuplicatesWithTagging[3]([Ljava.io.File;@268277c0, 4, 2, {mylib=[0, 2, 0, 0, 1, 1, 0.5, 0]}) | 0.334s | passed |
| testMarkDuplicatesSparkMarkingAllDuplicatesWithTagging[4]([Ljava.io.File;@214b183a, 4, 2, {mylib=[0, 2, 0, 0, 1, 1, 0.5, 0]}) | 0.336s | passed |
| testMarkDuplicatesSparkMarkingAllDuplicatesWithTagging[5]([Ljava.io.File;@620f7c57, 94, 8, {mylib=[0, 2, 0, 0, 1, 1, 0.5, 0], Solexa-16419=[4, 4, 4, 0, 0, 0, 0.0, 0], Solexa-16416=[2, 2, 2, 0, 0, 0, 0.0, 0], Solexa-16404=[3, 9, 3, 0, 2, 0, 0.190476, 17], Solexa-16406=[1, 10, 1, 0, 0, 0, 0.0, 0], Solexa-16412=[3, 6, 3, 0, 1, 0, 0.133333, 15]}) | 1.141s | passed |
| testMarkDuplicatesSparkMarkingAllDuplicatesWithTagging[6]([Ljava.io.File;@7981d081, 94, 8, {mylib=[0, 2, 0, 0, 1, 1, 0.5, 0], Solexa-16419=[4, 4, 4, 0, 0, 0, 0.0, 0], Solexa-16416=[2, 2, 2, 0, 0, 0, 0.0, 0], Solexa-16404=[3, 9, 3, 0, 2, 0, 0.190476, 17], Solexa-16406=[1, 10, 1, 0, 0, 0, 0.0, 0], Solexa-16412=[3, 6, 3, 0, 1, 0, 0.133333, 15]}) | 1.184s | passed |
| testMarkDuplicatesSparkMarkingOpticalDuplicatesWithTagging[0]([Ljava.io.File;@353055fd, 20, 0, {Solexa-16419=[0, 3, 0, 0, 0, 0, 0.0, 0], Solexa-16416=[0, 1, 0, 0, 0, 0, 0.0, 0], Solexa-16404=[0, 3, 0, 0, 0, 0, 0.0, 0], Solexa-16406=[0, 1, 0, 0, 0, 0, 0.0, 0], Solexa-16412=[0, 1, 0, 0, 0, 0, 0.0, 0]}) | 0.365s | passed |
| testMarkDuplicatesSparkMarkingOpticalDuplicatesWithTagging[1]([Ljava.io.File;@6b84006a, 90, 6, {Solexa-16419=[4, 4, 4, 0, 0, 0, 0.0, 0], Solexa-16416=[2, 2, 2, 0, 0, 0, 0.0, 0], Solexa-16404=[3, 9, 3, 0, 2, 0, 0.190476, 17], Solexa-16406=[1, 10, 1, 0, 0, 0, 0.0, 0], Solexa-16412=[3, 6, 3, 0, 1, 0, 0.133333, 15]}) | 0.354s | passed |
| testMarkDuplicatesSparkMarkingOpticalDuplicatesWithTagging[2]([Ljava.io.File;@6912d6, 90, 6, {Solexa-16419=[4, 4, 4, 0, 0, 0, 0.0, 0], Solexa-16416=[2, 2, 2, 0, 0, 0, 0.0, 0], Solexa-16404=[3, 9, 3, 0, 2, 0, 0.190476, 17], Solexa-16406=[1, 10, 1, 0, 0, 0, 0.0, 0], Solexa-16412=[3, 6, 3, 0, 1, 0, 0.133333, 15]}) | 0.348s | passed |
| testMarkDuplicatesSparkMarkingOpticalDuplicatesWithTagging[3]([Ljava.io.File;@58657403, 4, 2, {mylib=[0, 2, 0, 0, 1, 1, 0.5, 0]}) | 0.316s | passed |
| testMarkDuplicatesSparkMarkingOpticalDuplicatesWithTagging[4]([Ljava.io.File;@fbf5f08, 4, 2, {mylib=[0, 2, 0, 0, 1, 1, 0.5, 0]}) | 0.325s | passed |
| testMarkDuplicatesSparkMarkingOpticalDuplicatesWithTagging[5]([Ljava.io.File;@e582b99, 94, 8, {mylib=[0, 2, 0, 0, 1, 1, 0.5, 0], Solexa-16419=[4, 4, 4, 0, 0, 0, 0.0, 0], Solexa-16416=[2, 2, 2, 0, 0, 0, 0.0, 0], Solexa-16404=[3, 9, 3, 0, 2, 0, 0.190476, 17], Solexa-16406=[1, 10, 1, 0, 0, 0, 0.0, 0], Solexa-16412=[3, 6, 3, 0, 1, 0, 0.133333, 15]}) | 1.173s | passed |
| testMarkDuplicatesSparkMarkingOpticalDuplicatesWithTagging[6]([Ljava.io.File;@5608868f, 94, 8, {mylib=[0, 2, 0, 0, 1, 1, 0.5, 0], Solexa-16419=[4, 4, 4, 0, 0, 0, 0.0, 0], Solexa-16416=[2, 2, 2, 0, 0, 0, 0.0, 0], Solexa-16404=[3, 9, 3, 0, 2, 0, 0.190476, 17], Solexa-16406=[1, 10, 1, 0, 0, 0, 0.0, 0], Solexa-16412=[3, 6, 3, 0, 1, 0, 0.133333, 15]}) | 1.153s | passed |
| testMatePairFirstUnmapped | 0.293s | passed |
| testMatePairSecondUnmapped | 0.291s | passed |
| testNoReadGroupInRead | 0.229s | passed |
| testNonExistantReadGroupInRead | 0.209s | passed |
| testNullOpticalDuplicates | 0.318s | passed |
| testOpticalDuplicateClusterOneEndSamePositionOneCluster | 0.307s | passed |
| testOpticalDuplicateClusterSamePositionNoOpticalDuplicatesWithinPixelDistance | 0.291s | passed |
| testOpticalDuplicateClusterSamePositionNoOpticalDuplicates[0](RUNID:7:1203:2886:82292, RUNID:7:1205:3886:16834) | 0.305s | passed |
| testOpticalDuplicateClusterSamePositionNoOpticalDuplicates[10](RUNID:7:1203:6886:16756, RUNID:7:1203:4886:26756) | 0.296s | passed |
| testOpticalDuplicateClusterSamePositionNoOpticalDuplicates[11](RUNID:7:1203:2886:34756, RUNID:7:1203:2886:36756) | 0.297s | passed |
| testOpticalDuplicateClusterSamePositionNoOpticalDuplicates[12](RUNID:7:1203:2886:35756, RUNID:7:1203:2886:36756) | 0.308s | passed |
| testOpticalDuplicateClusterSamePositionNoOpticalDuplicates[13](RUNID:7:1203:2886:37756, RUNID:7:1203:2886:36756) | 0.293s | passed |
| testOpticalDuplicateClusterSamePositionNoOpticalDuplicates[14](RUNID:7:1203:2886:38756, RUNID:7:1203:2886:36756) | 0.298s | passed |
| testOpticalDuplicateClusterSamePositionNoOpticalDuplicates[15](HK3T5CCXX160204:3:1112:11586:37067, HK3T5CCXX160204:3:1112:11586:32144) | 0.294s | passed |
| testOpticalDuplicateClusterSamePositionNoOpticalDuplicates[1](RUNID:7:1203:2886:16756, RUNID:7:1205:3886:16756) | 0.300s | passed |
| testOpticalDuplicateClusterSamePositionNoOpticalDuplicates[2](RUNID:7:1204:2886:16756, RUNID:7:1205:3886:16756) | 0.296s | passed |
| testOpticalDuplicateClusterSamePositionNoOpticalDuplicates[3](RUNID:7:1205:2886:16756, RUNID:7:1205:3886:16756) | 0.304s | passed |
| testOpticalDuplicateClusterSamePositionNoOpticalDuplicates[4](RUNID:7:1206:2886:16756, RUNID:7:1205:3886:16756) | 0.305s | passed |
| testOpticalDuplicateClusterSamePositionNoOpticalDuplicates[5](RUNID:7:1207:2886:16756, RUNID:7:1205:3886:16756) | 0.296s | passed |
| testOpticalDuplicateClusterSamePositionNoOpticalDuplicates[6](RUNID:7:1203:2886:16756, RUNID:7:1203:4886:26756) | 0.292s | passed |
| testOpticalDuplicateClusterSamePositionNoOpticalDuplicates[7](RUNID:7:1203:3886:16756, RUNID:7:1203:4886:26756) | 0.288s | passed |
| testOpticalDuplicateClusterSamePositionNoOpticalDuplicates[8](RUNID:7:1203:4886:16756, RUNID:7:1203:4886:26756) | 0.305s | passed |
| testOpticalDuplicateClusterSamePositionNoOpticalDuplicates[9](RUNID:7:1203:5886:16756, RUNID:7:1203:4886:26756) | 0.292s | passed |
| testOpticalDuplicateClusterSamePositionOneOpticalDuplicatesWithinPixelDistance | 0.311s | passed |
| testOpticalDuplicateClustersAddingSecondEndFirstSameCoordinate | 0.300s | passed |
| testOpticalDuplicateFinding | 0.293s | passed |
| testOpticalDuplicateFindingPxelDistance | 0.290s | passed |
| testOpticalDuplicatesAndPCRDuplicatesOrientationDifference | 0.306s | passed |
| testOpticalDuplicatesDifferentReadGroups | 0.291s | passed |
| testOpticalDuplicatesTheSameReadGroup | 0.304s | passed |
| testOpticalDuplicatesTiebrokenByPhysicalLocationNotStartPosition[0](RUNID:7:1203:2886:82292, RUNID:7:1205:3886:16834) | 0.298s | passed |
| testOpticalDuplicatesTiebrokenByPhysicalLocationNotStartPosition[10](RUNID:7:1203:6886:16756, RUNID:7:1203:4886:26756) | 0.308s | passed |
| testOpticalDuplicatesTiebrokenByPhysicalLocationNotStartPosition[11](RUNID:7:1203:2886:34756, RUNID:7:1203:2886:36756) | 0.297s | passed |
| testOpticalDuplicatesTiebrokenByPhysicalLocationNotStartPosition[12](RUNID:7:1203:2886:35756, RUNID:7:1203:2886:36756) | 0.302s | passed |
| testOpticalDuplicatesTiebrokenByPhysicalLocationNotStartPosition[13](RUNID:7:1203:2886:37756, RUNID:7:1203:2886:36756) | 0.293s | passed |
| testOpticalDuplicatesTiebrokenByPhysicalLocationNotStartPosition[14](RUNID:7:1203:2886:38756, RUNID:7:1203:2886:36756) | 0.299s | passed |
| testOpticalDuplicatesTiebrokenByPhysicalLocationNotStartPosition[15](HK3T5CCXX160204:3:1112:11586:37067, HK3T5CCXX160204:3:1112:11586:32144) | 0.327s | passed |
| testOpticalDuplicatesTiebrokenByPhysicalLocationNotStartPosition[1](RUNID:7:1203:2886:16756, RUNID:7:1205:3886:16756) | 0.314s | passed |
| testOpticalDuplicatesTiebrokenByPhysicalLocationNotStartPosition[2](RUNID:7:1204:2886:16756, RUNID:7:1205:3886:16756) | 0.294s | passed |
| testOpticalDuplicatesTiebrokenByPhysicalLocationNotStartPosition[3](RUNID:7:1205:2886:16756, RUNID:7:1205:3886:16756) | 0.290s | passed |
| testOpticalDuplicatesTiebrokenByPhysicalLocationNotStartPosition[4](RUNID:7:1206:2886:16756, RUNID:7:1205:3886:16756) | 0.298s | passed |
| testOpticalDuplicatesTiebrokenByPhysicalLocationNotStartPosition[5](RUNID:7:1207:2886:16756, RUNID:7:1205:3886:16756) | 0.292s | passed |
| testOpticalDuplicatesTiebrokenByPhysicalLocationNotStartPosition[6](RUNID:7:1203:2886:16756, RUNID:7:1203:4886:26756) | 0.301s | passed |
| testOpticalDuplicatesTiebrokenByPhysicalLocationNotStartPosition[7](RUNID:7:1203:3886:16756, RUNID:7:1203:4886:26756) | 0.299s | passed |
| testOpticalDuplicatesTiebrokenByPhysicalLocationNotStartPosition[8](RUNID:7:1203:4886:16756, RUNID:7:1203:4886:26756) | 0.288s | passed |
| testOpticalDuplicatesTiebrokenByPhysicalLocationNotStartPosition[9](RUNID:7:1203:5886:16756, RUNID:7:1203:4886:26756) | 0.296s | passed |
| testPathologicalOrderingAtTheSamePosition | 0.308s | passed |
| testReadSameStartPositionOrientationOverride | 0.297s | passed |
| testReadsHaveSameStartPositionButDifferentChromosomeNonEquivalence | 0.294s | passed |
| testSecondEndIsBeforeFirstInCoordinate | 0.305s | passed |
| testSingleMappedFragment | 0.296s | passed |
| testSingleMappedFragmentAndSingleMappedPair | 0.289s | passed |
| testSingleMappedFragmentAndTwoMappedPairs | 0.295s | passed |
| testSingleMappedPair | 0.303s | passed |
| testSingleUnmappedFragment | 0.283s | passed |
| testSingleUnmappedPair | 0.290s | passed |
| testStackOverFlowPairSetSwap | 0.289s | passed |
| testSupplementaryReadUnmappedMate | 0.301s | passed |
| testThreeGroupsOnDifferentChromosomesOfThreeMappedPairs | 0.311s | passed |
| testThreeMappedPairs | 0.294s | passed |
| testThreeMappedPairsWithMatchingSecondMate | 0.291s | passed |
| testTwoGroupsOnDifferentChromosomesOfThreeMappedPairs | 0.287s | passed |
| testTwoGroupsOnDifferentChromosomesOfTwoFragments | 0.307s | passed |
| testTwoGroupsOnDifferentChromosomesOfTwoMappedPairs | 0.292s | passed |
| testTwoMappedFragments | 0.296s | passed |
| testTwoMappedPairWithSamePosition | 0.294s | passed |
| testTwoMappedPairWithSamePositionDifferentStrands | 0.292s | passed |
| testTwoMappedPairWithSamePositionDifferentStrands2 | 0.300s | passed |
| testTwoMappedPairs | 0.292s | passed |
| testTwoMappedPairsAndMappedSecondaryFragment | 0.287s | passed |
| testTwoMappedPairsAndTerminalUnmappedFragment | 0.293s | passed |
| testTwoMappedPairsAndTerminalUnmappedPair | 0.300s | passed |
| testTwoMappedPairsMatesSoftClipped | 0.286s | passed |
| testTwoMappedPairsWithOppositeOrientations | 0.285s | passed |
| testTwoMappedPairsWithOppositeOrientationsNumberTwo | 0.291s | passed |
| testTwoMappedPairsWithSoftClipping | 0.339s | passed |
| testTwoMappedPairsWithSoftClippingBoth | 0.290s | passed |
| testTwoMappedPairsWithSoftClippingFirstOfPairOnly | 0.298s | passed |
| testTwoMappedPairsWithSoftClippingFirstOfPairOnlyNoMateCigar | 0.285s | passed |
| testTwoMappedPairsWithSupplementaryReads[0](true, true, true) | 0.284s | passed |
| testTwoMappedPairsWithSupplementaryReads[1](true, false, true) | 0.299s | passed |
| testTwoMappedPairsWithSupplementaryReads[2](false, true, true) | 0.286s | passed |
| testTwoMappedPairsWithSupplementaryReads[3](true, true, false) | 0.293s | passed |
| testTwoMappedPairsWithSupplementaryReads[4](true, false, false) | 0.285s | passed |
| testTwoMappedPairsWithSupplementaryReads[5](false, true, false) | 0.302s | passed |
| testTwoUnmappedFragments | 0.285s | passed |
Standard error
[May 27, 2025 at 3:45:20 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.01 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:45:20 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:45:21 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.01 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:45:22 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.03 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:45:24 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.02 minutes. Runtime.totalMemory()=1426063360 15:45:24.040 INFO MarkDuplicatesSpark - ------------------------------------------------------------ 15:45:24.040 INFO MarkDuplicatesSpark - org.broadinstitute.hellbender.tools.spark.transforms.markduplicates vUnavailable 15:45:24.040 INFO MarkDuplicatesSpark - For support and documentation go to https://software.broadinstitute.org/gatk/ 15:45:24.040 INFO MarkDuplicatesSpark - Executing as runner@pkrvmf6wy0o8zjz on Linux v6.11.0-1014-azure amd64 15:45:24.040 INFO MarkDuplicatesSpark - Java runtime: OpenJDK 64-Bit Server VM v17.0.6+10 15:45:24.040 INFO MarkDuplicatesSpark - Start Date/Time: May 27, 2025 at 3:45:24 PM UTC 15:45:24.040 INFO MarkDuplicatesSpark - ------------------------------------------------------------ 15:45:24.040 INFO MarkDuplicatesSpark - ------------------------------------------------------------ 15:45:24.040 INFO MarkDuplicatesSpark - HTSJDK Defaults.COMPRESSION_LEVEL : 2 15:45:24.040 INFO MarkDuplicatesSpark - HTSJDK Defaults.USE_ASYNC_IO_READ_FOR_SAMTOOLS : false 15:45:24.040 INFO MarkDuplicatesSpark - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_SAMTOOLS : true 15:45:24.041 INFO MarkDuplicatesSpark - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_TRIBBLE : false 15:45:24.041 INFO MarkDuplicatesSpark - Deflater: IntelDeflater 15:45:24.041 INFO MarkDuplicatesSpark - Inflater: IntelInflater 15:45:24.041 INFO MarkDuplicatesSpark - GCS max retries/reopens: 20 15:45:24.041 INFO MarkDuplicatesSpark - Requester pays: disabled 15:45:24.041 INFO MarkDuplicatesSpark - Initializing engine 15:45:24.041 INFO MarkDuplicatesSpark - Done initializing engine 15:45:24.041 INFO MarkDuplicatesSpark - Spark verbosity set to INFO (see --spark-verbosity argument) 15:45:24.043 INFO MemoryStore - Block broadcast_1371 stored as values in memory (estimated size 306.3 KiB, free 1903.7 MiB) 15:45:24.050 INFO MemoryStore - Block broadcast_1371_piece0 stored as bytes in memory (estimated size 64.4 KiB, free 1903.7 MiB) 15:45:24.050 INFO BlockManagerInfo - Added broadcast_1371_piece0 in memory on localhost:43633 (size: 64.4 KiB, free: 1919.0 MiB) 15:45:24.051 INFO SparkContext - Created broadcast 1371 from newAPIHadoopFile at PathSplitSource.java:96 15:45:24.070 INFO MemoryStore - Block broadcast_1372 stored as values in memory (estimated size 306.3 KiB, free 1903.4 MiB) 15:45:24.077 INFO MemoryStore - Block broadcast_1372_piece0 stored as bytes in memory (estimated size 64.4 KiB, free 1903.3 MiB) 15:45:24.077 INFO BlockManagerInfo - Added broadcast_1372_piece0 in memory on localhost:43633 (size: 64.4 KiB, free: 1919.0 MiB) 15:45:24.077 INFO SparkContext - Created broadcast 1372 from newAPIHadoopFile at PathSplitSource.java:96 15:45:24.110 INFO SparkContext - Starting job: collect at SparkUtils.java:205 15:45:24.111 INFO FileInputFormat - Total input files to process : 1 15:45:24.112 INFO DAGScheduler - Registering RDD 3483 (mapToPair at SparkUtils.java:161) as input to shuffle 100 15:45:24.112 INFO DAGScheduler - Got job 423 (collect at SparkUtils.java:205) with 1 output partitions 15:45:24.112 INFO DAGScheduler - Final stage: ResultStage 735 (collect at SparkUtils.java:205) 15:45:24.112 INFO DAGScheduler - Parents of final stage: List(ShuffleMapStage 734) 15:45:24.112 INFO DAGScheduler - Missing parents: List(ShuffleMapStage 734) 15:45:24.113 INFO DAGScheduler - Submitting ShuffleMapStage 734 (MapPartitionsRDD[3483] at mapToPair at SparkUtils.java:161), which has no missing parents 15:45:24.129 INFO MemoryStore - Block broadcast_1373 stored as values in memory (estimated size 460.4 KiB, free 1902.8 MiB) 15:45:24.132 INFO MemoryStore - Block broadcast_1373_piece0 stored as bytes in memory (estimated size 202.5 KiB, free 1902.6 MiB) 15:45:24.132 INFO BlockManagerInfo - Added broadcast_1373_piece0 in memory on localhost:43633 (size: 202.5 KiB, free: 1918.8 MiB) 15:45:24.132 INFO SparkContext - Created broadcast 1373 from broadcast at DAGScheduler.scala:1580 15:45:24.133 INFO DAGScheduler - Submitting 1 missing tasks from ShuffleMapStage 734 (MapPartitionsRDD[3483] at mapToPair at SparkUtils.java:161) (first 15 tasks are for partitions Vector(0)) 15:45:24.133 INFO TaskSchedulerImpl - Adding task set 734.0 with 1 tasks resource profile 0 15:45:24.134 INFO TaskSetManager - Starting task 0.0 in stage 734.0 (TID 676) (localhost, executor driver, partition 0, PROCESS_LOCAL, 9735 bytes) 15:45:24.135 INFO Executor - Running task 0.0 in stage 734.0 (TID 676) 15:45:24.164 INFO NewHadoopRDD - Input split: file:/home/runner/work/gatk/gatk/src/test/resources/org/broadinstitute/hellbender/tools/walkers/MarkDuplicatesGATK/mdOrderBug.bam:0+1507 15:45:24.168 INFO Executor - Finished task 0.0 in stage 734.0 (TID 676). 1148 bytes result sent to driver 15:45:24.168 INFO TaskSetManager - Finished task 0.0 in stage 734.0 (TID 676) in 35 ms on localhost (executor driver) (1/1) 15:45:24.168 INFO TaskSchedulerImpl - Removed TaskSet 734.0, whose tasks have all completed, from pool 15:45:24.169 INFO DAGScheduler - ShuffleMapStage 734 (mapToPair at SparkUtils.java:161) finished in 0.055 s 15:45:24.169 INFO DAGScheduler - looking for newly runnable stages 15:45:24.169 INFO DAGScheduler - running: HashSet() 15:45:24.169 INFO DAGScheduler - waiting: HashSet(ResultStage 735) 15:45:24.169 INFO DAGScheduler - failed: HashSet() 15:45:24.169 INFO DAGScheduler - Submitting ResultStage 735 (MapPartitionsRDD[3486] at mapPartitions at SparkUtils.java:188), which has no missing parents 15:45:24.170 INFO MemoryStore - Block broadcast_1374 stored as values in memory (estimated size 7.4 KiB, free 1902.6 MiB) 15:45:24.170 INFO MemoryStore - Block broadcast_1374_piece0 stored as bytes in memory (estimated size 4.1 KiB, free 1902.6 MiB) 15:45:24.170 INFO BlockManagerInfo - Added broadcast_1374_piece0 in memory on localhost:43633 (size: 4.1 KiB, free: 1918.8 MiB) 15:45:24.171 INFO SparkContext - Created broadcast 1374 from broadcast at DAGScheduler.scala:1580 15:45:24.171 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 735 (MapPartitionsRDD[3486] at mapPartitions at SparkUtils.java:188) (first 15 tasks are for partitions Vector(0)) 15:45:24.171 INFO TaskSchedulerImpl - Adding task set 735.0 with 1 tasks resource profile 0 15:45:24.171 INFO TaskSetManager - Starting task 0.0 in stage 735.0 (TID 677) (localhost, executor driver, partition 0, NODE_LOCAL, 9376 bytes) 15:45:24.172 INFO Executor - Running task 0.0 in stage 735.0 (TID 677) 15:45:24.173 INFO ShuffleBlockFetcherIterator - Getting 1 (405.0 B) non-empty blocks including 1 (405.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks 15:45:24.173 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms 15:45:24.175 INFO Executor - Finished task 0.0 in stage 735.0 (TID 677). 2238 bytes result sent to driver 15:45:24.175 INFO TaskSetManager - Finished task 0.0 in stage 735.0 (TID 677) in 4 ms on localhost (executor driver) (1/1) 15:45:24.175 INFO TaskSchedulerImpl - Removed TaskSet 735.0, whose tasks have all completed, from pool 15:45:24.175 INFO DAGScheduler - ResultStage 735 (collect at SparkUtils.java:205) finished in 0.006 s 15:45:24.175 INFO DAGScheduler - Job 423 is finished. Cancelling potential speculative or zombie tasks for this job 15:45:24.175 INFO TaskSchedulerImpl - Killing all running tasks in stage 735: Stage finished 15:45:24.176 INFO DAGScheduler - Job 423 finished: collect at SparkUtils.java:205, took 0.065050 s 15:45:24.182 INFO MemoryStore - Block broadcast_1375 stored as values in memory (estimated size 1632.0 B, free 1902.6 MiB) 15:45:24.182 INFO MemoryStore - Block broadcast_1375_piece0 stored as bytes in memory (estimated size 145.0 B, free 1902.6 MiB) 15:45:24.182 INFO BlockManagerInfo - Added broadcast_1375_piece0 in memory on localhost:43633 (size: 145.0 B, free: 1918.8 MiB) 15:45:24.182 INFO SparkContext - Created broadcast 1375 from broadcast at MarkDuplicatesSparkUtils.java:126 15:45:24.183 INFO MemoryStore - Block broadcast_1376 stored as values in memory (estimated size 304.0 B, free 1902.6 MiB) 15:45:24.183 INFO MemoryStore - Block broadcast_1376_piece0 stored as bytes in memory (estimated size 37.0 B, free 1902.6 MiB) 15:45:24.183 INFO BlockManagerInfo - Added broadcast_1376_piece0 in memory on localhost:43633 (size: 37.0 B, free: 1918.8 MiB) 15:45:24.183 INFO SparkContext - Created broadcast 1376 from broadcast at MarkDuplicatesSparkUtils.java:127 15:45:24.195 INFO SparkContext - Starting job: collectAsMap at MarkDuplicatesSparkUtils.java:542 15:45:24.196 INFO DAGScheduler - Registering RDD 3495 (flatMapToPair at MarkDuplicatesSparkUtils.java:130) as input to shuffle 103 15:45:24.196 INFO DAGScheduler - Registering RDD 3499 (mapToPair at MarkDuplicatesSpark.java:217) as input to shuffle 102 15:45:24.196 INFO DAGScheduler - Registering RDD 3503 (mapToPair at MarkDuplicatesSparkUtils.java:497) as input to shuffle 101 15:45:24.196 INFO DAGScheduler - Got job 424 (collectAsMap at MarkDuplicatesSparkUtils.java:542) with 1 output partitions 15:45:24.196 INFO DAGScheduler - Final stage: ResultStage 740 (collectAsMap at MarkDuplicatesSparkUtils.java:542) 15:45:24.196 INFO DAGScheduler - Parents of final stage: List(ShuffleMapStage 739) 15:45:24.197 INFO DAGScheduler - Missing parents: List(ShuffleMapStage 739) 15:45:24.197 INFO DAGScheduler - Submitting ShuffleMapStage 737 (MapPartitionsRDD[3495] at flatMapToPair at MarkDuplicatesSparkUtils.java:130), which has no missing parents 15:45:24.198 INFO MemoryStore - Block broadcast_1377 stored as values in memory (estimated size 24.2 KiB, free 1902.6 MiB) 15:45:24.198 INFO MemoryStore - Block broadcast_1377_piece0 stored as bytes in memory (estimated size 12.6 KiB, free 1902.6 MiB) 15:45:24.198 INFO BlockManagerInfo - Added broadcast_1377_piece0 in memory on localhost:43633 (size: 12.6 KiB, free: 1918.8 MiB) 15:45:24.198 INFO SparkContext - Created broadcast 1377 from broadcast at DAGScheduler.scala:1580 15:45:24.199 INFO DAGScheduler - Submitting 1 missing tasks from ShuffleMapStage 737 (MapPartitionsRDD[3495] at flatMapToPair at MarkDuplicatesSparkUtils.java:130) (first 15 tasks are for partitions Vector(0)) 15:45:24.199 INFO TaskSchedulerImpl - Adding task set 737.0 with 1 tasks resource profile 0 15:45:24.199 INFO TaskSetManager - Starting task 0.0 in stage 737.0 (TID 678) (localhost, executor driver, partition 0, NODE_LOCAL, 9890 bytes) 15:45:24.199 INFO Executor - Running task 0.0 in stage 737.0 (TID 678) 15:45:24.201 INFO ShuffleBlockFetcherIterator - Getting 1 (405.0 B) non-empty blocks including 1 (405.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks 15:45:24.201 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms 15:45:24.204 INFO Executor - Finished task 0.0 in stage 737.0 (TID 678). 1922 bytes result sent to driver 15:45:24.204 INFO TaskSetManager - Finished task 0.0 in stage 737.0 (TID 678) in 5 ms on localhost (executor driver) (1/1) 15:45:24.204 INFO TaskSchedulerImpl - Removed TaskSet 737.0, whose tasks have all completed, from pool 15:45:24.205 INFO DAGScheduler - ShuffleMapStage 737 (flatMapToPair at MarkDuplicatesSparkUtils.java:130) finished in 0.008 s 15:45:24.205 INFO DAGScheduler - looking for newly runnable stages 15:45:24.205 INFO DAGScheduler - running: HashSet() 15:45:24.205 INFO DAGScheduler - waiting: HashSet(ShuffleMapStage 738, ShuffleMapStage 739, ResultStage 740) 15:45:24.205 INFO DAGScheduler - failed: HashSet() 15:45:24.205 INFO DAGScheduler - Submitting ShuffleMapStage 738 (MapPartitionsRDD[3499] at mapToPair at MarkDuplicatesSpark.java:217), which has no missing parents 15:45:24.206 INFO MemoryStore - Block broadcast_1378 stored as values in memory (estimated size 28.3 KiB, free 1902.6 MiB) 15:45:24.206 INFO MemoryStore - Block broadcast_1378_piece0 stored as bytes in memory (estimated size 14.8 KiB, free 1902.6 MiB) 15:45:24.206 INFO BlockManagerInfo - Added broadcast_1378_piece0 in memory on localhost:43633 (size: 14.8 KiB, free: 1918.8 MiB) 15:45:24.206 INFO SparkContext - Created broadcast 1378 from broadcast at DAGScheduler.scala:1580 15:45:24.207 INFO DAGScheduler - Submitting 1 missing tasks from ShuffleMapStage 738 (MapPartitionsRDD[3499] at mapToPair at MarkDuplicatesSpark.java:217) (first 15 tasks are for partitions Vector(0)) 15:45:24.207 INFO TaskSchedulerImpl - Adding task set 738.0 with 1 tasks resource profile 0 15:45:24.207 INFO TaskSetManager - Starting task 0.0 in stage 738.0 (TID 679) (localhost, executor driver, partition 0, NODE_LOCAL, 9365 bytes) 15:45:24.207 INFO Executor - Running task 0.0 in stage 738.0 (TID 679) 15:45:24.209 INFO ShuffleBlockFetcherIterator - Getting 1 (129.0 B) non-empty blocks including 1 (129.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks 15:45:24.209 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms 15:45:24.213 INFO Executor - Finished task 0.0 in stage 738.0 (TID 679). 1922 bytes result sent to driver 15:45:24.213 INFO TaskSetManager - Finished task 0.0 in stage 738.0 (TID 679) in 6 ms on localhost (executor driver) (1/1) 15:45:24.213 INFO TaskSchedulerImpl - Removed TaskSet 738.0, whose tasks have all completed, from pool 15:45:24.213 INFO DAGScheduler - ShuffleMapStage 738 (mapToPair at MarkDuplicatesSpark.java:217) finished in 0.008 s 15:45:24.213 INFO DAGScheduler - looking for newly runnable stages 15:45:24.213 INFO DAGScheduler - running: HashSet() 15:45:24.213 INFO DAGScheduler - waiting: HashSet(ShuffleMapStage 739, ResultStage 740) 15:45:24.213 INFO DAGScheduler - failed: HashSet() 15:45:24.214 INFO DAGScheduler - Submitting ShuffleMapStage 739 (MapPartitionsRDD[3503] at mapToPair at MarkDuplicatesSparkUtils.java:497), which has no missing parents 15:45:24.214 INFO MemoryStore - Block broadcast_1379 stored as values in memory (estimated size 19.0 KiB, free 1902.5 MiB) 15:45:24.215 INFO MemoryStore - Block broadcast_1379_piece0 stored as bytes in memory (estimated size 9.7 KiB, free 1902.5 MiB) 15:45:24.215 INFO BlockManagerInfo - Added broadcast_1379_piece0 in memory on localhost:43633 (size: 9.7 KiB, free: 1918.7 MiB) 15:45:24.215 INFO SparkContext - Created broadcast 1379 from broadcast at DAGScheduler.scala:1580 15:45:24.215 INFO DAGScheduler - Submitting 1 missing tasks from ShuffleMapStage 739 (MapPartitionsRDD[3503] at mapToPair at MarkDuplicatesSparkUtils.java:497) (first 15 tasks are for partitions Vector(0)) 15:45:24.215 INFO TaskSchedulerImpl - Adding task set 739.0 with 1 tasks resource profile 0 15:45:24.216 INFO TaskSetManager - Starting task 0.0 in stage 739.0 (TID 680) (localhost, executor driver, partition 0, NODE_LOCAL, 9948 bytes) 15:45:24.216 INFO Executor - Running task 0.0 in stage 739.0 (TID 680) 15:45:24.218 INFO ShuffleBlockFetcherIterator - Getting 1 (405.0 B) non-empty blocks including 1 (405.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks 15:45:24.218 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms 15:45:24.219 INFO ShuffleBlockFetcherIterator - Getting 1 (117.0 B) non-empty blocks including 1 (117.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks 15:45:24.219 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms 15:45:24.221 INFO Executor - Finished task 0.0 in stage 739.0 (TID 680). 1922 bytes result sent to driver 15:45:24.221 INFO TaskSetManager - Finished task 0.0 in stage 739.0 (TID 680) in 5 ms on localhost (executor driver) (1/1) 15:45:24.221 INFO TaskSchedulerImpl - Removed TaskSet 739.0, whose tasks have all completed, from pool 15:45:24.221 INFO DAGScheduler - ShuffleMapStage 739 (mapToPair at MarkDuplicatesSparkUtils.java:497) finished in 0.007 s 15:45:24.222 INFO DAGScheduler - looking for newly runnable stages 15:45:24.222 INFO DAGScheduler - running: HashSet() 15:45:24.222 INFO DAGScheduler - waiting: HashSet(ResultStage 740) 15:45:24.222 INFO DAGScheduler - failed: HashSet() 15:45:24.222 INFO DAGScheduler - Submitting ResultStage 740 (MapPartitionsRDD[3505] at mapValues at MarkDuplicatesSparkUtils.java:519), which has no missing parents 15:45:24.223 INFO MemoryStore - Block broadcast_1380 stored as values in memory (estimated size 20.2 KiB, free 1902.5 MiB) 15:45:24.223 INFO MemoryStore - Block broadcast_1380_piece0 stored as bytes in memory (estimated size 10.6 KiB, free 1902.5 MiB) 15:45:24.223 INFO BlockManagerInfo - Added broadcast_1380_piece0 in memory on localhost:43633 (size: 10.6 KiB, free: 1918.7 MiB) 15:45:24.224 INFO SparkContext - Created broadcast 1380 from broadcast at DAGScheduler.scala:1580 15:45:24.224 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 740 (MapPartitionsRDD[3505] at mapValues at MarkDuplicatesSparkUtils.java:519) (first 15 tasks are for partitions Vector(0)) 15:45:24.224 INFO TaskSchedulerImpl - Adding task set 740.0 with 1 tasks resource profile 0 15:45:24.224 INFO TaskSetManager - Starting task 0.0 in stage 740.0 (TID 681) (localhost, executor driver, partition 0, NODE_LOCAL, 9376 bytes) 15:45:24.225 INFO Executor - Running task 0.0 in stage 740.0 (TID 681) 15:45:24.226 INFO ShuffleBlockFetcherIterator - Getting 1 (117.0 B) non-empty blocks including 1 (117.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks 15:45:24.226 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms 15:45:24.228 INFO Executor - Finished task 0.0 in stage 740.0 (TID 681). 1901 bytes result sent to driver 15:45:24.228 INFO TaskSetManager - Finished task 0.0 in stage 740.0 (TID 681) in 4 ms on localhost (executor driver) (1/1) 15:45:24.228 INFO TaskSchedulerImpl - Removed TaskSet 740.0, whose tasks have all completed, from pool 15:45:24.228 INFO DAGScheduler - ResultStage 740 (collectAsMap at MarkDuplicatesSparkUtils.java:542) finished in 0.006 s 15:45:24.228 INFO DAGScheduler - Job 424 is finished. Cancelling potential speculative or zombie tasks for this job 15:45:24.229 INFO TaskSchedulerImpl - Killing all running tasks in stage 740: Stage finished 15:45:24.229 INFO DAGScheduler - Job 424 finished: collectAsMap at MarkDuplicatesSparkUtils.java:542, took 0.033171 s 15:45:24.240 INFO MemoryStore - Block broadcast_1381 stored as values in memory (estimated size 20.3 KiB, free 1902.5 MiB) 15:45:24.240 INFO MemoryStore - Block broadcast_1381_piece0 stored as bytes in memory (estimated size 1850.0 B, free 1902.5 MiB) 15:45:24.240 INFO BlockManagerInfo - Added broadcast_1381_piece0 in memory on localhost:43633 (size: 1850.0 B, free: 1918.7 MiB) 15:45:24.240 INFO SparkContext - Created broadcast 1381 from broadcast at ReadsSparkSink.java:133 15:45:24.241 INFO MemoryStore - Block broadcast_1382 stored as values in memory (estimated size 20.4 KiB, free 1902.5 MiB) 15:45:24.242 INFO MemoryStore - Block broadcast_1382_piece0 stored as bytes in memory (estimated size 1850.0 B, free 1902.5 MiB) 15:45:24.242 INFO BlockManagerInfo - Added broadcast_1382_piece0 in memory on localhost:43633 (size: 1850.0 B, free: 1918.7 MiB) 15:45:24.242 INFO SparkContext - Created broadcast 1382 from broadcast at BamSink.java:76 15:45:24.244 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory 15:45:24.244 INFO FileOutputCommitter - File Output Committer Algorithm version is 1 15:45:24.244 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false 15:45:24.261 INFO SparkContext - Starting job: runJob at SparkHadoopWriter.scala:83 15:45:24.262 INFO DAGScheduler - Registering RDD 3507 (mapToPair at SparkUtils.java:161) as input to shuffle 104 15:45:24.262 INFO DAGScheduler - Got job 425 (runJob at SparkHadoopWriter.scala:83) with 1 output partitions 15:45:24.262 INFO DAGScheduler - Final stage: ResultStage 745 (runJob at SparkHadoopWriter.scala:83) 15:45:24.262 INFO DAGScheduler - Parents of final stage: List(ShuffleMapStage 744) 15:45:24.262 INFO DAGScheduler - Missing parents: List(ShuffleMapStage 744) 15:45:24.262 INFO DAGScheduler - Submitting ShuffleMapStage 744 (MapPartitionsRDD[3507] at mapToPair at SparkUtils.java:161), which has no missing parents 15:45:24.263 INFO MemoryStore - Block broadcast_1383 stored as values in memory (estimated size 18.1 KiB, free 1902.4 MiB) 15:45:24.264 INFO MemoryStore - Block broadcast_1383_piece0 stored as bytes in memory (estimated size 9.5 KiB, free 1902.4 MiB) 15:45:24.264 INFO BlockManagerInfo - Added broadcast_1383_piece0 in memory on localhost:43633 (size: 9.5 KiB, free: 1918.7 MiB) 15:45:24.264 INFO SparkContext - Created broadcast 1383 from broadcast at DAGScheduler.scala:1580 15:45:24.265 INFO DAGScheduler - Submitting 1 missing tasks from ShuffleMapStage 744 (MapPartitionsRDD[3507] at mapToPair at SparkUtils.java:161) (first 15 tasks are for partitions Vector(0)) 15:45:24.265 INFO TaskSchedulerImpl - Adding task set 744.0 with 1 tasks resource profile 0 15:45:24.265 INFO TaskSetManager - Starting task 0.0 in stage 744.0 (TID 682) (localhost, executor driver, partition 0, NODE_LOCAL, 9948 bytes) 15:45:24.265 INFO Executor - Running task 0.0 in stage 744.0 (TID 682) 15:45:24.267 INFO ShuffleBlockFetcherIterator - Getting 1 (405.0 B) non-empty blocks including 1 (405.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks 15:45:24.267 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms 15:45:24.269 INFO ShuffleBlockFetcherIterator - Getting 1 (117.0 B) non-empty blocks including 1 (117.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks 15:45:24.269 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms 15:45:24.270 INFO Executor - Finished task 0.0 in stage 744.0 (TID 682). 1922 bytes result sent to driver 15:45:24.271 INFO TaskSetManager - Finished task 0.0 in stage 744.0 (TID 682) in 6 ms on localhost (executor driver) (1/1) 15:45:24.271 INFO TaskSchedulerImpl - Removed TaskSet 744.0, whose tasks have all completed, from pool 15:45:24.271 INFO DAGScheduler - ShuffleMapStage 744 (mapToPair at SparkUtils.java:161) finished in 0.008 s 15:45:24.271 INFO DAGScheduler - looking for newly runnable stages 15:45:24.271 INFO DAGScheduler - running: HashSet() 15:45:24.271 INFO DAGScheduler - waiting: HashSet(ResultStage 745) 15:45:24.271 INFO DAGScheduler - failed: HashSet() 15:45:24.271 INFO DAGScheduler - Submitting ResultStage 745 (MapPartitionsRDD[3512] at mapToPair at BamSink.java:91), which has no missing parents 15:45:24.278 INFO MemoryStore - Block broadcast_1384 stored as values in memory (estimated size 163.7 KiB, free 1902.3 MiB) 15:45:24.279 INFO MemoryStore - Block broadcast_1384_piece0 stored as bytes in memory (estimated size 73.9 KiB, free 1902.2 MiB) 15:45:24.279 INFO BlockManagerInfo - Added broadcast_1384_piece0 in memory on localhost:43633 (size: 73.9 KiB, free: 1918.7 MiB) 15:45:24.279 INFO SparkContext - Created broadcast 1384 from broadcast at DAGScheduler.scala:1580 15:45:24.279 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 745 (MapPartitionsRDD[3512] at mapToPair at BamSink.java:91) (first 15 tasks are for partitions Vector(0)) 15:45:24.279 INFO TaskSchedulerImpl - Adding task set 745.0 with 1 tasks resource profile 0 15:45:24.280 INFO TaskSetManager - Starting task 0.0 in stage 745.0 (TID 683) (localhost, executor driver, partition 0, NODE_LOCAL, 9376 bytes) 15:45:24.280 INFO Executor - Running task 0.0 in stage 745.0 (TID 683) 15:45:24.286 INFO ShuffleBlockFetcherIterator - Getting 1 (405.0 B) non-empty blocks including 1 (405.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks 15:45:24.286 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms 15:45:24.288 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory 15:45:24.288 INFO FileOutputCommitter - File Output Committer Algorithm version is 1 15:45:24.288 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false 15:45:24.289 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory 15:45:24.289 INFO FileOutputCommitter - File Output Committer Algorithm version is 1 15:45:24.289 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false 15:45:24.309 INFO FileOutputCommitter - Saved output of task 'attempt_202505271545246647553100259646976_3512_r_000000_0' to file:/tmp/local15669066122832454348/markdups14933085452718983455.bam.parts/_temporary/0/task_202505271545246647553100259646976_3512_r_000000 15:45:24.310 INFO SparkHadoopMapRedUtil - attempt_202505271545246647553100259646976_3512_r_000000_0: Committed. Elapsed time: 0 ms. 15:45:24.310 INFO Executor - Finished task 0.0 in stage 745.0 (TID 683). 1858 bytes result sent to driver 15:45:24.310 INFO TaskSetManager - Finished task 0.0 in stage 745.0 (TID 683) in 30 ms on localhost (executor driver) (1/1) 15:45:24.311 INFO TaskSchedulerImpl - Removed TaskSet 745.0, whose tasks have all completed, from pool 15:45:24.311 INFO DAGScheduler - ResultStage 745 (runJob at SparkHadoopWriter.scala:83) finished in 0.039 s 15:45:24.311 INFO DAGScheduler - Job 425 is finished. Cancelling potential speculative or zombie tasks for this job 15:45:24.311 INFO TaskSchedulerImpl - Killing all running tasks in stage 745: Stage finished 15:45:24.311 INFO DAGScheduler - Job 425 finished: runJob at SparkHadoopWriter.scala:83, took 0.049811 s 15:45:24.311 INFO SparkHadoopWriter - Start to commit write Job job_202505271545246647553100259646976_3512. 15:45:24.316 INFO SparkHadoopWriter - Write Job job_202505271545246647553100259646976_3512 committed. Elapsed time: 4 ms. 15:45:24.327 INFO HadoopFileSystemWrapper - Concatenating 3 parts to /tmp/local15669066122832454348/markdups14933085452718983455.bam 15:45:24.331 INFO HadoopFileSystemWrapper - Concatenating to /tmp/local15669066122832454348/markdups14933085452718983455.bam done 15:45:24.331 INFO IndexFileMerger - Merging .sbi files in temp directory /tmp/local15669066122832454348/markdups14933085452718983455.bam.parts/ to /tmp/local15669066122832454348/markdups14933085452718983455.bam.sbi 15:45:24.335 INFO IndexFileMerger - Done merging .sbi files 15:45:24.335 INFO IndexFileMerger - Merging .bai files in temp directory /tmp/local15669066122832454348/markdups14933085452718983455.bam.parts/ to /tmp/local15669066122832454348/markdups14933085452718983455.bam.bai 15:45:24.339 INFO IndexFileMerger - Done merging .bai files 15:45:24.339 INFO MarkDuplicatesSpark - Shutting down engine [May 27, 2025 at 3:45:24 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.01 minutes. Runtime.totalMemory()=1426063360 15:45:24.348 INFO MarkDuplicatesSpark - ------------------------------------------------------------ 15:45:24.348 INFO MarkDuplicatesSpark - org.broadinstitute.hellbender.tools.spark.transforms.markduplicates vUnavailable 15:45:24.348 INFO MarkDuplicatesSpark - For support and documentation go to https://software.broadinstitute.org/gatk/ 15:45:24.348 INFO MarkDuplicatesSpark - Executing as runner@pkrvmf6wy0o8zjz on Linux v6.11.0-1014-azure amd64 15:45:24.348 INFO MarkDuplicatesSpark - Java runtime: OpenJDK 64-Bit Server VM v17.0.6+10 15:45:24.348 INFO MarkDuplicatesSpark - Start Date/Time: May 27, 2025 at 3:45:24 PM UTC 15:45:24.348 INFO MarkDuplicatesSpark - ------------------------------------------------------------ 15:45:24.348 INFO MarkDuplicatesSpark - ------------------------------------------------------------ 15:45:24.349 INFO MarkDuplicatesSpark - HTSJDK Defaults.COMPRESSION_LEVEL : 2 15:45:24.349 INFO MarkDuplicatesSpark - HTSJDK Defaults.USE_ASYNC_IO_READ_FOR_SAMTOOLS : false 15:45:24.349 INFO MarkDuplicatesSpark - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_SAMTOOLS : true 15:45:24.349 INFO MarkDuplicatesSpark - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_TRIBBLE : false 15:45:24.349 INFO MarkDuplicatesSpark - Deflater: IntelDeflater 15:45:24.349 INFO MarkDuplicatesSpark - Inflater: IntelInflater 15:45:24.349 INFO MarkDuplicatesSpark - GCS max retries/reopens: 20 15:45:24.349 INFO MarkDuplicatesSpark - Requester pays: disabled 15:45:24.349 INFO MarkDuplicatesSpark - Initializing engine 15:45:24.349 INFO MarkDuplicatesSpark - Done initializing engine 15:45:24.349 INFO MarkDuplicatesSpark - Spark verbosity set to INFO (see --spark-verbosity argument) 15:45:24.351 INFO MemoryStore - Block broadcast_1385 stored as values in memory (estimated size 306.3 KiB, free 1901.9 MiB) 15:45:24.358 INFO MemoryStore - Block broadcast_1385_piece0 stored as bytes in memory (estimated size 64.4 KiB, free 1901.8 MiB) 15:45:24.358 INFO BlockManagerInfo - Added broadcast_1385_piece0 in memory on localhost:43633 (size: 64.4 KiB, free: 1918.6 MiB) 15:45:24.358 INFO SparkContext - Created broadcast 1385 from newAPIHadoopFile at PathSplitSource.java:96 15:45:24.377 INFO MemoryStore - Block broadcast_1386 stored as values in memory (estimated size 306.3 KiB, free 1901.5 MiB) 15:45:24.384 INFO MemoryStore - Block broadcast_1386_piece0 stored as bytes in memory (estimated size 64.4 KiB, free 1901.5 MiB) 15:45:24.384 INFO BlockManagerInfo - Added broadcast_1386_piece0 in memory on localhost:43633 (size: 64.4 KiB, free: 1918.5 MiB) 15:45:24.384 INFO SparkContext - Created broadcast 1386 from newAPIHadoopFile at PathSplitSource.java:96 15:45:24.416 INFO SparkContext - Starting job: collect at SparkUtils.java:205 15:45:24.416 INFO FileInputFormat - Total input files to process : 1 15:45:24.417 INFO DAGScheduler - Registering RDD 3526 (mapToPair at SparkUtils.java:161) as input to shuffle 105 15:45:24.417 INFO DAGScheduler - Got job 426 (collect at SparkUtils.java:205) with 1 output partitions 15:45:24.417 INFO DAGScheduler - Final stage: ResultStage 747 (collect at SparkUtils.java:205) 15:45:24.417 INFO DAGScheduler - Parents of final stage: List(ShuffleMapStage 746) 15:45:24.417 INFO DAGScheduler - Missing parents: List(ShuffleMapStage 746) 15:45:24.417 INFO DAGScheduler - Submitting ShuffleMapStage 746 (MapPartitionsRDD[3526] at mapToPair at SparkUtils.java:161), which has no missing parents 15:45:24.434 INFO MemoryStore - Block broadcast_1387 stored as values in memory (estimated size 460.4 KiB, free 1901.0 MiB) 15:45:24.436 INFO MemoryStore - Block broadcast_1387_piece0 stored as bytes in memory (estimated size 202.5 KiB, free 1900.8 MiB) 15:45:24.436 INFO BlockManagerInfo - Added broadcast_1387_piece0 in memory on localhost:43633 (size: 202.5 KiB, free: 1918.3 MiB) 15:45:24.436 INFO SparkContext - Created broadcast 1387 from broadcast at DAGScheduler.scala:1580 15:45:24.436 INFO DAGScheduler - Submitting 1 missing tasks from ShuffleMapStage 746 (MapPartitionsRDD[3526] at mapToPair at SparkUtils.java:161) (first 15 tasks are for partitions Vector(0)) 15:45:24.437 INFO TaskSchedulerImpl - Adding task set 746.0 with 1 tasks resource profile 0 15:45:24.437 INFO TaskSetManager - Starting task 0.0 in stage 746.0 (TID 684) (localhost, executor driver, partition 0, PROCESS_LOCAL, 9736 bytes) 15:45:24.437 INFO Executor - Running task 0.0 in stage 746.0 (TID 684) 15:45:24.467 INFO NewHadoopRDD - Input split: file:/home/runner/work/gatk/gatk/src/test/resources/org/broadinstitute/hellbender/tools/walkers/MarkDuplicatesGATK/mdOrderBug2.bam:0+1856 15:45:24.471 INFO Executor - Finished task 0.0 in stage 746.0 (TID 684). 1148 bytes result sent to driver 15:45:24.471 INFO TaskSetManager - Finished task 0.0 in stage 746.0 (TID 684) in 34 ms on localhost (executor driver) (1/1) 15:45:24.471 INFO TaskSchedulerImpl - Removed TaskSet 746.0, whose tasks have all completed, from pool 15:45:24.471 INFO DAGScheduler - ShuffleMapStage 746 (mapToPair at SparkUtils.java:161) finished in 0.054 s 15:45:24.472 INFO DAGScheduler - looking for newly runnable stages 15:45:24.472 INFO DAGScheduler - running: HashSet() 15:45:24.472 INFO DAGScheduler - waiting: HashSet(ResultStage 747) 15:45:24.472 INFO DAGScheduler - failed: HashSet() 15:45:24.472 INFO DAGScheduler - Submitting ResultStage 747 (MapPartitionsRDD[3529] at mapPartitions at SparkUtils.java:188), which has no missing parents 15:45:24.472 INFO MemoryStore - Block broadcast_1388 stored as values in memory (estimated size 7.4 KiB, free 1900.8 MiB) 15:45:24.473 INFO MemoryStore - Block broadcast_1388_piece0 stored as bytes in memory (estimated size 4.1 KiB, free 1900.8 MiB) 15:45:24.473 INFO BlockManagerInfo - Added broadcast_1388_piece0 in memory on localhost:43633 (size: 4.1 KiB, free: 1918.3 MiB) 15:45:24.473 INFO SparkContext - Created broadcast 1388 from broadcast at DAGScheduler.scala:1580 15:45:24.473 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 747 (MapPartitionsRDD[3529] at mapPartitions at SparkUtils.java:188) (first 15 tasks are for partitions Vector(0)) 15:45:24.473 INFO TaskSchedulerImpl - Adding task set 747.0 with 1 tasks resource profile 0 15:45:24.474 INFO TaskSetManager - Starting task 0.0 in stage 747.0 (TID 685) (localhost, executor driver, partition 0, NODE_LOCAL, 9376 bytes) 15:45:24.474 INFO Executor - Running task 0.0 in stage 747.0 (TID 685) 15:45:24.475 INFO ShuffleBlockFetcherIterator - Getting 1 (868.0 B) non-empty blocks including 1 (868.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks 15:45:24.475 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms 15:45:24.477 INFO Executor - Finished task 0.0 in stage 747.0 (TID 685). 2495 bytes result sent to driver 15:45:24.477 INFO TaskSetManager - Finished task 0.0 in stage 747.0 (TID 685) in 3 ms on localhost (executor driver) (1/1) 15:45:24.477 INFO TaskSchedulerImpl - Removed TaskSet 747.0, whose tasks have all completed, from pool 15:45:24.477 INFO DAGScheduler - ResultStage 747 (collect at SparkUtils.java:205) finished in 0.005 s 15:45:24.477 INFO DAGScheduler - Job 426 is finished. Cancelling potential speculative or zombie tasks for this job 15:45:24.477 INFO TaskSchedulerImpl - Killing all running tasks in stage 747: Stage finished 15:45:24.478 INFO DAGScheduler - Job 426 finished: collect at SparkUtils.java:205, took 0.061696 s 15:45:24.482 INFO MemoryStore - Block broadcast_1389 stored as values in memory (estimated size 1632.0 B, free 1900.8 MiB) 15:45:24.482 INFO MemoryStore - Block broadcast_1389_piece0 stored as bytes in memory (estimated size 145.0 B, free 1900.8 MiB) 15:45:24.482 INFO BlockManagerInfo - Added broadcast_1389_piece0 in memory on localhost:43633 (size: 145.0 B, free: 1918.3 MiB) 15:45:24.483 INFO SparkContext - Created broadcast 1389 from broadcast at MarkDuplicatesSparkUtils.java:126 15:45:24.483 INFO MemoryStore - Block broadcast_1390 stored as values in memory (estimated size 304.0 B, free 1900.8 MiB) 15:45:24.483 INFO MemoryStore - Block broadcast_1390_piece0 stored as bytes in memory (estimated size 37.0 B, free 1900.8 MiB) 15:45:24.483 INFO BlockManagerInfo - Added broadcast_1390_piece0 in memory on localhost:43633 (size: 37.0 B, free: 1918.3 MiB) 15:45:24.483 INFO SparkContext - Created broadcast 1390 from broadcast at MarkDuplicatesSparkUtils.java:127 15:45:24.496 INFO SparkContext - Starting job: collectAsMap at MarkDuplicatesSparkUtils.java:542 15:45:24.496 INFO DAGScheduler - Registering RDD 3538 (flatMapToPair at MarkDuplicatesSparkUtils.java:130) as input to shuffle 108 15:45:24.496 INFO DAGScheduler - Registering RDD 3542 (mapToPair at MarkDuplicatesSpark.java:217) as input to shuffle 107 15:45:24.496 INFO DAGScheduler - Registering RDD 3546 (mapToPair at MarkDuplicatesSparkUtils.java:497) as input to shuffle 106 15:45:24.496 INFO DAGScheduler - Got job 427 (collectAsMap at MarkDuplicatesSparkUtils.java:542) with 1 output partitions 15:45:24.496 INFO DAGScheduler - Final stage: ResultStage 752 (collectAsMap at MarkDuplicatesSparkUtils.java:542) 15:45:24.497 INFO DAGScheduler - Parents of final stage: List(ShuffleMapStage 751) 15:45:24.497 INFO DAGScheduler - Missing parents: List(ShuffleMapStage 751) 15:45:24.497 INFO DAGScheduler - Submitting ShuffleMapStage 749 (MapPartitionsRDD[3538] at flatMapToPair at MarkDuplicatesSparkUtils.java:130), which has no missing parents 15:45:24.498 INFO MemoryStore - Block broadcast_1391 stored as values in memory (estimated size 24.2 KiB, free 1900.8 MiB) 15:45:24.498 INFO MemoryStore - Block broadcast_1391_piece0 stored as bytes in memory (estimated size 12.6 KiB, free 1900.8 MiB) 15:45:24.498 INFO BlockManagerInfo - Added broadcast_1391_piece0 in memory on localhost:43633 (size: 12.6 KiB, free: 1918.3 MiB) 15:45:24.498 INFO SparkContext - Created broadcast 1391 from broadcast at DAGScheduler.scala:1580 15:45:24.499 INFO DAGScheduler - Submitting 1 missing tasks from ShuffleMapStage 749 (MapPartitionsRDD[3538] at flatMapToPair at MarkDuplicatesSparkUtils.java:130) (first 15 tasks are for partitions Vector(0)) 15:45:24.499 INFO TaskSchedulerImpl - Adding task set 749.0 with 1 tasks resource profile 0 15:45:24.499 INFO TaskSetManager - Starting task 0.0 in stage 749.0 (TID 686) (localhost, executor driver, partition 0, NODE_LOCAL, 9890 bytes) 15:45:24.499 INFO Executor - Running task 0.0 in stage 749.0 (TID 686) 15:45:24.501 INFO ShuffleBlockFetcherIterator - Getting 1 (868.0 B) non-empty blocks including 1 (868.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks 15:45:24.501 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms 15:45:24.504 INFO Executor - Finished task 0.0 in stage 749.0 (TID 686). 1922 bytes result sent to driver 15:45:24.504 INFO TaskSetManager - Finished task 0.0 in stage 749.0 (TID 686) in 5 ms on localhost (executor driver) (1/1) 15:45:24.504 INFO TaskSchedulerImpl - Removed TaskSet 749.0, whose tasks have all completed, from pool 15:45:24.504 INFO DAGScheduler - ShuffleMapStage 749 (flatMapToPair at MarkDuplicatesSparkUtils.java:130) finished in 0.007 s 15:45:24.504 INFO DAGScheduler - looking for newly runnable stages 15:45:24.504 INFO DAGScheduler - running: HashSet() 15:45:24.504 INFO DAGScheduler - waiting: HashSet(ResultStage 752, ShuffleMapStage 750, ShuffleMapStage 751) 15:45:24.504 INFO DAGScheduler - failed: HashSet() 15:45:24.504 INFO DAGScheduler - Submitting ShuffleMapStage 750 (MapPartitionsRDD[3542] at mapToPair at MarkDuplicatesSpark.java:217), which has no missing parents 15:45:24.505 INFO MemoryStore - Block broadcast_1392 stored as values in memory (estimated size 28.3 KiB, free 1900.7 MiB) 15:45:24.506 INFO MemoryStore - Block broadcast_1392_piece0 stored as bytes in memory (estimated size 14.8 KiB, free 1900.7 MiB) 15:45:24.506 INFO BlockManagerInfo - Added broadcast_1392_piece0 in memory on localhost:43633 (size: 14.8 KiB, free: 1918.3 MiB) 15:45:24.506 INFO SparkContext - Created broadcast 1392 from broadcast at DAGScheduler.scala:1580 15:45:24.506 INFO DAGScheduler - Submitting 1 missing tasks from ShuffleMapStage 750 (MapPartitionsRDD[3542] at mapToPair at MarkDuplicatesSpark.java:217) (first 15 tasks are for partitions Vector(0)) 15:45:24.506 INFO TaskSchedulerImpl - Adding task set 750.0 with 1 tasks resource profile 0 15:45:24.506 INFO TaskSetManager - Starting task 0.0 in stage 750.0 (TID 687) (localhost, executor driver, partition 0, NODE_LOCAL, 9365 bytes) 15:45:24.507 INFO Executor - Running task 0.0 in stage 750.0 (TID 687) 15:45:24.508 INFO ShuffleBlockFetcherIterator - Getting 1 (142.0 B) non-empty blocks including 1 (142.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks 15:45:24.508 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms 15:45:24.513 INFO Executor - Finished task 0.0 in stage 750.0 (TID 687). 1922 bytes result sent to driver 15:45:24.513 INFO TaskSetManager - Finished task 0.0 in stage 750.0 (TID 687) in 7 ms on localhost (executor driver) (1/1) 15:45:24.513 INFO TaskSchedulerImpl - Removed TaskSet 750.0, whose tasks have all completed, from pool 15:45:24.513 INFO DAGScheduler - ShuffleMapStage 750 (mapToPair at MarkDuplicatesSpark.java:217) finished in 0.008 s 15:45:24.513 INFO DAGScheduler - looking for newly runnable stages 15:45:24.513 INFO DAGScheduler - running: HashSet() 15:45:24.513 INFO DAGScheduler - waiting: HashSet(ResultStage 752, ShuffleMapStage 751) 15:45:24.513 INFO DAGScheduler - failed: HashSet() 15:45:24.513 INFO DAGScheduler - Submitting ShuffleMapStage 751 (MapPartitionsRDD[3546] at mapToPair at MarkDuplicatesSparkUtils.java:497), which has no missing parents 15:45:24.514 INFO MemoryStore - Block broadcast_1393 stored as values in memory (estimated size 19.0 KiB, free 1900.7 MiB) 15:45:24.515 INFO MemoryStore - Block broadcast_1393_piece0 stored as bytes in memory (estimated size 9.7 KiB, free 1900.7 MiB) 15:45:24.515 INFO BlockManagerInfo - Added broadcast_1393_piece0 in memory on localhost:43633 (size: 9.7 KiB, free: 1918.3 MiB) 15:45:24.515 INFO SparkContext - Created broadcast 1393 from broadcast at DAGScheduler.scala:1580 15:45:24.515 INFO DAGScheduler - Submitting 1 missing tasks from ShuffleMapStage 751 (MapPartitionsRDD[3546] at mapToPair at MarkDuplicatesSparkUtils.java:497) (first 15 tasks are for partitions Vector(0)) 15:45:24.515 INFO TaskSchedulerImpl - Adding task set 751.0 with 1 tasks resource profile 0 15:45:24.515 INFO TaskSetManager - Starting task 0.0 in stage 751.0 (TID 688) (localhost, executor driver, partition 0, NODE_LOCAL, 9948 bytes) 15:45:24.516 INFO Executor - Running task 0.0 in stage 751.0 (TID 688) 15:45:24.517 INFO ShuffleBlockFetcherIterator - Getting 1 (868.0 B) non-empty blocks including 1 (868.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks 15:45:24.517 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms 15:45:24.519 INFO ShuffleBlockFetcherIterator - Getting 1 (117.0 B) non-empty blocks including 1 (117.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks 15:45:24.519 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms 15:45:24.521 INFO Executor - Finished task 0.0 in stage 751.0 (TID 688). 1922 bytes result sent to driver 15:45:24.521 INFO TaskSetManager - Finished task 0.0 in stage 751.0 (TID 688) in 6 ms on localhost (executor driver) (1/1) 15:45:24.521 INFO TaskSchedulerImpl - Removed TaskSet 751.0, whose tasks have all completed, from pool 15:45:24.521 INFO DAGScheduler - ShuffleMapStage 751 (mapToPair at MarkDuplicatesSparkUtils.java:497) finished in 0.007 s 15:45:24.521 INFO DAGScheduler - looking for newly runnable stages 15:45:24.521 INFO DAGScheduler - running: HashSet() 15:45:24.521 INFO DAGScheduler - waiting: HashSet(ResultStage 752) 15:45:24.521 INFO DAGScheduler - failed: HashSet() 15:45:24.522 INFO DAGScheduler - Submitting ResultStage 752 (MapPartitionsRDD[3548] at mapValues at MarkDuplicatesSparkUtils.java:519), which has no missing parents 15:45:24.522 INFO MemoryStore - Block broadcast_1394 stored as values in memory (estimated size 20.2 KiB, free 1900.7 MiB) 15:45:24.523 INFO MemoryStore - Block broadcast_1394_piece0 stored as bytes in memory (estimated size 10.6 KiB, free 1900.7 MiB) 15:45:24.523 INFO BlockManagerInfo - Added broadcast_1394_piece0 in memory on localhost:43633 (size: 10.6 KiB, free: 1918.3 MiB) 15:45:24.523 INFO SparkContext - Created broadcast 1394 from broadcast at DAGScheduler.scala:1580 15:45:24.523 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 752 (MapPartitionsRDD[3548] at mapValues at MarkDuplicatesSparkUtils.java:519) (first 15 tasks are for partitions Vector(0)) 15:45:24.523 INFO TaskSchedulerImpl - Adding task set 752.0 with 1 tasks resource profile 0 15:45:24.524 INFO TaskSetManager - Starting task 0.0 in stage 752.0 (TID 689) (localhost, executor driver, partition 0, NODE_LOCAL, 9376 bytes) 15:45:24.524 INFO Executor - Running task 0.0 in stage 752.0 (TID 689) 15:45:24.525 INFO ShuffleBlockFetcherIterator - Getting 1 (142.0 B) non-empty blocks including 1 (142.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks 15:45:24.525 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms 15:45:24.527 INFO Executor - Finished task 0.0 in stage 752.0 (TID 689). 2053 bytes result sent to driver 15:45:24.528 INFO TaskSetManager - Finished task 0.0 in stage 752.0 (TID 689) in 5 ms on localhost (executor driver) (1/1) 15:45:24.528 INFO TaskSchedulerImpl - Removed TaskSet 752.0, whose tasks have all completed, from pool 15:45:24.528 INFO DAGScheduler - ResultStage 752 (collectAsMap at MarkDuplicatesSparkUtils.java:542) finished in 0.006 s 15:45:24.528 INFO DAGScheduler - Job 427 is finished. Cancelling potential speculative or zombie tasks for this job 15:45:24.528 INFO TaskSchedulerImpl - Killing all running tasks in stage 752: Stage finished 15:45:24.528 INFO DAGScheduler - Job 427 finished: collectAsMap at MarkDuplicatesSparkUtils.java:542, took 0.032550 s 15:45:24.539 INFO MemoryStore - Block broadcast_1395 stored as values in memory (estimated size 20.3 KiB, free 1900.7 MiB) 15:45:24.539 INFO MemoryStore - Block broadcast_1395_piece0 stored as bytes in memory (estimated size 1850.0 B, free 1900.7 MiB) 15:45:24.540 INFO BlockManagerInfo - Added broadcast_1395_piece0 in memory on localhost:43633 (size: 1850.0 B, free: 1918.3 MiB) 15:45:24.540 INFO SparkContext - Created broadcast 1395 from broadcast at ReadsSparkSink.java:133 15:45:24.540 INFO MemoryStore - Block broadcast_1396 stored as values in memory (estimated size 20.4 KiB, free 1900.6 MiB) 15:45:24.541 INFO MemoryStore - Block broadcast_1396_piece0 stored as bytes in memory (estimated size 1850.0 B, free 1900.6 MiB) 15:45:24.541 INFO BlockManagerInfo - Added broadcast_1396_piece0 in memory on localhost:43633 (size: 1850.0 B, free: 1918.3 MiB) 15:45:24.541 INFO SparkContext - Created broadcast 1396 from broadcast at BamSink.java:76 15:45:24.543 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory 15:45:24.543 INFO FileOutputCommitter - File Output Committer Algorithm version is 1 15:45:24.543 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false 15:45:24.560 INFO SparkContext - Starting job: runJob at SparkHadoopWriter.scala:83 15:45:24.561 INFO DAGScheduler - Registering RDD 3550 (mapToPair at SparkUtils.java:161) as input to shuffle 109 15:45:24.561 INFO DAGScheduler - Got job 428 (runJob at SparkHadoopWriter.scala:83) with 1 output partitions 15:45:24.561 INFO DAGScheduler - Final stage: ResultStage 757 (runJob at SparkHadoopWriter.scala:83) 15:45:24.561 INFO DAGScheduler - Parents of final stage: List(ShuffleMapStage 756) 15:45:24.561 INFO DAGScheduler - Missing parents: List(ShuffleMapStage 756) 15:45:24.561 INFO DAGScheduler - Submitting ShuffleMapStage 756 (MapPartitionsRDD[3550] at mapToPair at SparkUtils.java:161), which has no missing parents 15:45:24.562 INFO MemoryStore - Block broadcast_1397 stored as values in memory (estimated size 18.1 KiB, free 1900.6 MiB) 15:45:24.562 INFO MemoryStore - Block broadcast_1397_piece0 stored as bytes in memory (estimated size 9.5 KiB, free 1900.6 MiB) 15:45:24.563 INFO BlockManagerInfo - Added broadcast_1397_piece0 in memory on localhost:43633 (size: 9.5 KiB, free: 1918.3 MiB) 15:45:24.563 INFO SparkContext - Created broadcast 1397 from broadcast at DAGScheduler.scala:1580 15:45:24.563 INFO DAGScheduler - Submitting 1 missing tasks from ShuffleMapStage 756 (MapPartitionsRDD[3550] at mapToPair at SparkUtils.java:161) (first 15 tasks are for partitions Vector(0)) 15:45:24.563 INFO TaskSchedulerImpl - Adding task set 756.0 with 1 tasks resource profile 0 15:45:24.564 INFO TaskSetManager - Starting task 0.0 in stage 756.0 (TID 690) (localhost, executor driver, partition 0, NODE_LOCAL, 9948 bytes) 15:45:24.564 INFO Executor - Running task 0.0 in stage 756.0 (TID 690) 15:45:24.565 INFO ShuffleBlockFetcherIterator - Getting 1 (868.0 B) non-empty blocks including 1 (868.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks 15:45:24.565 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms 15:45:24.567 INFO ShuffleBlockFetcherIterator - Getting 1 (117.0 B) non-empty blocks including 1 (117.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks 15:45:24.567 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms 15:45:24.568 INFO Executor - Finished task 0.0 in stage 756.0 (TID 690). 1922 bytes result sent to driver 15:45:24.569 INFO TaskSetManager - Finished task 0.0 in stage 756.0 (TID 690) in 6 ms on localhost (executor driver) (1/1) 15:45:24.569 INFO TaskSchedulerImpl - Removed TaskSet 756.0, whose tasks have all completed, from pool 15:45:24.569 INFO DAGScheduler - ShuffleMapStage 756 (mapToPair at SparkUtils.java:161) finished in 0.008 s 15:45:24.569 INFO DAGScheduler - looking for newly runnable stages 15:45:24.569 INFO DAGScheduler - running: HashSet() 15:45:24.569 INFO DAGScheduler - waiting: HashSet(ResultStage 757) 15:45:24.569 INFO DAGScheduler - failed: HashSet() 15:45:24.569 INFO DAGScheduler - Submitting ResultStage 757 (MapPartitionsRDD[3555] at mapToPair at BamSink.java:91), which has no missing parents 15:45:24.580 INFO MemoryStore - Block broadcast_1398 stored as values in memory (estimated size 163.7 KiB, free 1900.4 MiB) 15:45:24.581 INFO MemoryStore - Block broadcast_1398_piece0 stored as bytes in memory (estimated size 73.9 KiB, free 1900.4 MiB) 15:45:24.581 INFO BlockManagerInfo - Added broadcast_1398_piece0 in memory on localhost:43633 (size: 73.9 KiB, free: 1918.2 MiB) 15:45:24.581 INFO SparkContext - Created broadcast 1398 from broadcast at DAGScheduler.scala:1580 15:45:24.582 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 757 (MapPartitionsRDD[3555] at mapToPair at BamSink.java:91) (first 15 tasks are for partitions Vector(0)) 15:45:24.582 INFO TaskSchedulerImpl - Adding task set 757.0 with 1 tasks resource profile 0 15:45:24.582 INFO TaskSetManager - Starting task 0.0 in stage 757.0 (TID 691) (localhost, executor driver, partition 0, NODE_LOCAL, 9376 bytes) 15:45:24.582 INFO Executor - Running task 0.0 in stage 757.0 (TID 691) 15:45:24.586 INFO ShuffleBlockFetcherIterator - Getting 1 (868.0 B) non-empty blocks including 1 (868.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks 15:45:24.586 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms 15:45:24.588 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory 15:45:24.588 INFO FileOutputCommitter - File Output Committer Algorithm version is 1 15:45:24.588 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false 15:45:24.589 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory 15:45:24.589 INFO FileOutputCommitter - File Output Committer Algorithm version is 1 15:45:24.589 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false 15:45:24.607 INFO FileOutputCommitter - Saved output of task 'attempt_202505271545245982239647797102678_3555_r_000000_0' to file:/tmp/local15669066122832454348/markdups4885684187670374696.bam.parts/_temporary/0/task_202505271545245982239647797102678_3555_r_000000 15:45:24.607 INFO SparkHadoopMapRedUtil - attempt_202505271545245982239647797102678_3555_r_000000_0: Committed. Elapsed time: 0 ms. 15:45:24.607 INFO Executor - Finished task 0.0 in stage 757.0 (TID 691). 1858 bytes result sent to driver 15:45:24.607 INFO TaskSetManager - Finished task 0.0 in stage 757.0 (TID 691) in 25 ms on localhost (executor driver) (1/1) 15:45:24.608 INFO TaskSchedulerImpl - Removed TaskSet 757.0, whose tasks have all completed, from pool 15:45:24.608 INFO DAGScheduler - ResultStage 757 (runJob at SparkHadoopWriter.scala:83) finished in 0.039 s 15:45:24.608 INFO DAGScheduler - Job 428 is finished. Cancelling potential speculative or zombie tasks for this job 15:45:24.608 INFO TaskSchedulerImpl - Killing all running tasks in stage 757: Stage finished 15:45:24.608 INFO DAGScheduler - Job 428 finished: runJob at SparkHadoopWriter.scala:83, took 0.047635 s 15:45:24.608 INFO SparkHadoopWriter - Start to commit write Job job_202505271545245982239647797102678_3555. 15:45:24.613 INFO SparkHadoopWriter - Write Job job_202505271545245982239647797102678_3555 committed. Elapsed time: 4 ms. 15:45:24.623 INFO HadoopFileSystemWrapper - Concatenating 3 parts to /tmp/local15669066122832454348/markdups4885684187670374696.bam 15:45:24.627 INFO HadoopFileSystemWrapper - Concatenating to /tmp/local15669066122832454348/markdups4885684187670374696.bam done 15:45:24.627 INFO IndexFileMerger - Merging .sbi files in temp directory /tmp/local15669066122832454348/markdups4885684187670374696.bam.parts/ to /tmp/local15669066122832454348/markdups4885684187670374696.bam.sbi 15:45:24.631 INFO IndexFileMerger - Done merging .sbi files 15:45:24.631 INFO IndexFileMerger - Merging .bai files in temp directory /tmp/local15669066122832454348/markdups4885684187670374696.bam.parts/ to /tmp/local15669066122832454348/markdups4885684187670374696.bam.bai 15:45:24.635 INFO IndexFileMerger - Done merging .bai files 15:45:24.635 INFO MarkDuplicatesSpark - Shutting down engine [May 27, 2025 at 3:45:24 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:45:24 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:45:25 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:45:25 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.01 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:45:26 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:45:26 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:45:26 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:45:27 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.02 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:45:29 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.02 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:45:29 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.01 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:45:29 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.01 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:45:30 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.01 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:45:30 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:45:30 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:45:32 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.02 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:45:33 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.02 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:45:33 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:45:34 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:45:34 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:45:34 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:45:35 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:45:36 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.02 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:45:37 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.02 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:45:37 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:45:38 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:45:38 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:45:38 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:45:39 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:45:40 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.02 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:45:41 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.02 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:45:41 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.01 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:45:42 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:45:42 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:45:42 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:45:43 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:45:44 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.02 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:45:45 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.02 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:45:45 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:45:46 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.01 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:45:48 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.03 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:45:49 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 15:45:49.036 INFO MarkDuplicatesSpark - ------------------------------------------------------------ 15:45:49.036 INFO MarkDuplicatesSpark - org.broadinstitute.hellbender.tools.spark.transforms.markduplicates vUnavailable 15:45:49.036 INFO MarkDuplicatesSpark - For support and documentation go to https://software.broadinstitute.org/gatk/ 15:45:49.036 INFO MarkDuplicatesSpark - Executing as runner@pkrvmf6wy0o8zjz on Linux v6.11.0-1014-azure amd64 15:45:49.036 INFO MarkDuplicatesSpark - Java runtime: OpenJDK 64-Bit Server VM v17.0.6+10 15:45:49.036 INFO MarkDuplicatesSpark - Start Date/Time: May 27, 2025 at 3:45:49 PM UTC 15:45:49.036 INFO MarkDuplicatesSpark - ------------------------------------------------------------ 15:45:49.036 INFO MarkDuplicatesSpark - ------------------------------------------------------------ 15:45:49.036 INFO MarkDuplicatesSpark - HTSJDK Defaults.COMPRESSION_LEVEL : 2 15:45:49.037 INFO MarkDuplicatesSpark - HTSJDK Defaults.USE_ASYNC_IO_READ_FOR_SAMTOOLS : false 15:45:49.037 INFO MarkDuplicatesSpark - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_SAMTOOLS : true 15:45:49.037 INFO MarkDuplicatesSpark - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_TRIBBLE : false 15:45:49.037 INFO MarkDuplicatesSpark - Deflater: IntelDeflater 15:45:49.037 INFO MarkDuplicatesSpark - Inflater: IntelInflater 15:45:49.037 INFO MarkDuplicatesSpark - GCS max retries/reopens: 20 15:45:49.037 INFO MarkDuplicatesSpark - Requester pays: disabled 15:45:49.037 INFO MarkDuplicatesSpark - Initializing engine 15:45:49.037 INFO MarkDuplicatesSpark - Done initializing engine 15:45:49.037 INFO MarkDuplicatesSpark - Spark verbosity set to INFO (see --spark-verbosity argument) 15:45:49.039 INFO MemoryStore - Block broadcast_1975 stored as values in memory (estimated size 306.3 KiB, free 1916.9 MiB) 15:45:49.046 INFO MemoryStore - Block broadcast_1975_piece0 stored as bytes in memory (estimated size 64.4 KiB, free 1916.8 MiB) 15:45:49.046 INFO BlockManagerInfo - Added broadcast_1975_piece0 in memory on localhost:43633 (size: 64.4 KiB, free: 1919.3 MiB) 15:45:49.046 INFO SparkContext - Created broadcast 1975 from newAPIHadoopFile at PathSplitSource.java:96 15:45:49.066 INFO MemoryStore - Block broadcast_1976 stored as values in memory (estimated size 306.3 KiB, free 1916.5 MiB) 15:45:49.073 INFO MemoryStore - Block broadcast_1976_piece0 stored as bytes in memory (estimated size 64.4 KiB, free 1916.5 MiB) 15:45:49.073 INFO BlockManagerInfo - Added broadcast_1976_piece0 in memory on localhost:43633 (size: 64.4 KiB, free: 1919.2 MiB) 15:45:49.073 INFO SparkContext - Created broadcast 1976 from newAPIHadoopFile at PathSplitSource.java:96 15:45:49.106 INFO SparkContext - Starting job: collect at SparkUtils.java:205 15:45:49.106 INFO FileInputFormat - Total input files to process : 1 15:45:49.107 INFO DAGScheduler - Registering RDD 5447 (mapToPair at SparkUtils.java:161) as input to shuffle 300 15:45:49.107 INFO DAGScheduler - Got job 559 (collect at SparkUtils.java:205) with 1 output partitions 15:45:49.107 INFO DAGScheduler - Final stage: ResultStage 1219 (collect at SparkUtils.java:205) 15:45:49.107 INFO DAGScheduler - Parents of final stage: List(ShuffleMapStage 1218) 15:45:49.107 INFO DAGScheduler - Missing parents: List(ShuffleMapStage 1218) 15:45:49.107 INFO DAGScheduler - Submitting ShuffleMapStage 1218 (MapPartitionsRDD[5447] at mapToPair at SparkUtils.java:161), which has no missing parents 15:45:49.124 INFO MemoryStore - Block broadcast_1977 stored as values in memory (estimated size 493.8 KiB, free 1916.0 MiB) 15:45:49.127 INFO MemoryStore - Block broadcast_1977_piece0 stored as bytes in memory (estimated size 210.1 KiB, free 1915.8 MiB) 15:45:49.127 INFO BlockManagerInfo - Added broadcast_1977_piece0 in memory on localhost:43633 (size: 210.1 KiB, free: 1919.0 MiB) 15:45:49.127 INFO SparkContext - Created broadcast 1977 from broadcast at DAGScheduler.scala:1580 15:45:49.127 INFO DAGScheduler - Submitting 1 missing tasks from ShuffleMapStage 1218 (MapPartitionsRDD[5447] at mapToPair at SparkUtils.java:161) (first 15 tasks are for partitions Vector(0)) 15:45:49.127 INFO TaskSchedulerImpl - Adding task set 1218.0 with 1 tasks resource profile 0 15:45:49.128 INFO TaskSetManager - Starting task 0.0 in stage 1218.0 (TID 1062) (localhost, executor driver, partition 0, PROCESS_LOCAL, 9758 bytes) 15:45:49.128 INFO Executor - Running task 0.0 in stage 1218.0 (TID 1062) 15:45:49.160 INFO BlockManagerInfo - Removed broadcast_1973_piece0 on localhost:43633 in memory (size: 8.0 KiB, free: 1919.0 MiB) 15:45:49.161 INFO BlockManagerInfo - Removed broadcast_1969_piece0 on localhost:43633 in memory (size: 13.2 KiB, free: 1919.0 MiB) 15:45:49.161 INFO BlockManagerInfo - Removed broadcast_1974_piece0 on localhost:43633 in memory (size: 72.1 KiB, free: 1919.1 MiB) 15:45:49.162 INFO BlockManagerInfo - Removed broadcast_1967_piece0 on localhost:43633 in memory (size: 31.0 B, free: 1919.1 MiB) 15:45:49.162 INFO BlockManagerInfo - Removed broadcast_1964_piece0 on localhost:43633 in memory (size: 201.2 KiB, free: 1919.3 MiB) 15:45:49.163 INFO BlockManagerInfo - Removed broadcast_1961_piece0 on localhost:43633 in memory (size: 72.1 KiB, free: 1919.4 MiB) 15:45:49.163 INFO BlockManagerInfo - Removed broadcast_1966_piece0 on localhost:43633 in memory (size: 19.0 B, free: 1919.4 MiB) 15:45:49.163 INFO BlockManagerInfo - Removed broadcast_1968_piece0 on localhost:43633 in memory (size: 11.2 KiB, free: 1919.4 MiB) 15:45:49.164 INFO BlockManagerInfo - Removed broadcast_1972_piece0 on localhost:43633 in memory (size: 663.0 B, free: 1919.4 MiB) 15:45:49.164 INFO BlockManagerInfo - Removed broadcast_1958_piece0 on localhost:43633 in memory (size: 9.1 KiB, free: 1919.4 MiB) 15:45:49.165 INFO BlockManagerInfo - Removed broadcast_1975_piece0 on localhost:43633 in memory (size: 64.4 KiB, free: 1919.4 MiB) 15:45:49.165 INFO BlockManagerInfo - Removed broadcast_1960_piece0 on localhost:43633 in memory (size: 8.0 KiB, free: 1919.5 MiB) 15:45:49.165 INFO BlockManagerInfo - Removed broadcast_1965_piece0 on localhost:43633 in memory (size: 4.1 KiB, free: 1919.5 MiB) 15:45:49.165 INFO BlockManagerInfo - Removed broadcast_1963_piece0 on localhost:43633 in memory (size: 64.3 KiB, free: 1919.5 MiB) 15:45:49.166 INFO BlockManagerInfo - Removed broadcast_1962_piece0 on localhost:43633 in memory (size: 64.3 KiB, free: 1919.6 MiB) 15:45:49.166 INFO BlockManagerInfo - Removed broadcast_1970_piece0 on localhost:43633 in memory (size: 8.5 KiB, free: 1919.6 MiB) 15:45:49.167 INFO BlockManagerInfo - Removed broadcast_1971_piece0 on localhost:43633 in memory (size: 9.1 KiB, free: 1919.6 MiB) 15:45:49.167 INFO BlockManagerInfo - Removed broadcast_1959_piece0 on localhost:43633 in memory (size: 663.0 B, free: 1919.6 MiB) 15:45:49.170 INFO NewHadoopRDD - Input split: file:/home/runner/work/gatk/gatk/src/test/resources/org/broadinstitute/hellbender/tools/walkers/MarkDuplicatesGATK/example.chr1.1-1K.unmarkedDups.bam:0+13412 15:45:49.177 INFO Executor - Finished task 0.0 in stage 1218.0 (TID 1062). 1191 bytes result sent to driver 15:45:49.178 INFO TaskSetManager - Finished task 0.0 in stage 1218.0 (TID 1062) in 50 ms on localhost (executor driver) (1/1) 15:45:49.178 INFO DAGScheduler - ShuffleMapStage 1218 (mapToPair at SparkUtils.java:161) finished in 0.070 s 15:45:49.178 INFO DAGScheduler - looking for newly runnable stages 15:45:49.178 INFO DAGScheduler - running: HashSet() 15:45:49.178 INFO DAGScheduler - waiting: HashSet(ResultStage 1219) 15:45:49.178 INFO DAGScheduler - failed: HashSet() 15:45:49.178 INFO TaskSchedulerImpl - Removed TaskSet 1218.0, whose tasks have all completed, from pool 15:45:49.178 INFO DAGScheduler - Submitting ResultStage 1219 (MapPartitionsRDD[5450] at mapPartitions at SparkUtils.java:188), which has no missing parents 15:45:49.179 INFO MemoryStore - Block broadcast_1978 stored as values in memory (estimated size 7.4 KiB, free 1918.2 MiB) 15:45:49.179 INFO MemoryStore - Block broadcast_1978_piece0 stored as bytes in memory (estimated size 4.1 KiB, free 1918.2 MiB) 15:45:49.179 INFO BlockManagerInfo - Added broadcast_1978_piece0 in memory on localhost:43633 (size: 4.1 KiB, free: 1919.6 MiB) 15:45:49.179 INFO SparkContext - Created broadcast 1978 from broadcast at DAGScheduler.scala:1580 15:45:49.180 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 1219 (MapPartitionsRDD[5450] at mapPartitions at SparkUtils.java:188) (first 15 tasks are for partitions Vector(0)) 15:45:49.180 INFO TaskSchedulerImpl - Adding task set 1219.0 with 1 tasks resource profile 0 15:45:49.180 INFO TaskSetManager - Starting task 0.0 in stage 1219.0 (TID 1063) (localhost, executor driver, partition 0, NODE_LOCAL, 9376 bytes) 15:45:49.180 INFO Executor - Running task 0.0 in stage 1219.0 (TID 1063) 15:45:49.181 INFO ShuffleBlockFetcherIterator - Getting 1 (16.3 KiB) non-empty blocks including 1 (16.3 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks 15:45:49.181 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms 15:45:49.185 INFO Executor - Finished task 0.0 in stage 1219.0 (TID 1063). 2329 bytes result sent to driver 15:45:49.186 INFO TaskSetManager - Finished task 0.0 in stage 1219.0 (TID 1063) in 6 ms on localhost (executor driver) (1/1) 15:45:49.186 INFO TaskSchedulerImpl - Removed TaskSet 1219.0, whose tasks have all completed, from pool 15:45:49.186 INFO DAGScheduler - ResultStage 1219 (collect at SparkUtils.java:205) finished in 0.008 s 15:45:49.186 INFO DAGScheduler - Job 559 is finished. Cancelling potential speculative or zombie tasks for this job 15:45:49.186 INFO TaskSchedulerImpl - Killing all running tasks in stage 1219: Stage finished 15:45:49.186 INFO DAGScheduler - Job 559 finished: collect at SparkUtils.java:205, took 0.079875 s 15:45:49.192 INFO MemoryStore - Block broadcast_1979 stored as values in memory (estimated size 7.8 KiB, free 1918.2 MiB) 15:45:49.192 INFO MemoryStore - Block broadcast_1979_piece0 stored as bytes in memory (estimated size 540.0 B, free 1918.2 MiB) 15:45:49.192 INFO BlockManagerInfo - Added broadcast_1979_piece0 in memory on localhost:43633 (size: 540.0 B, free: 1919.6 MiB) 15:45:49.193 INFO SparkContext - Created broadcast 1979 from broadcast at MarkDuplicatesSparkUtils.java:126 15:45:49.193 INFO MemoryStore - Block broadcast_1980 stored as values in memory (estimated size 2.6 KiB, free 1918.2 MiB) 15:45:49.193 INFO MemoryStore - Block broadcast_1980_piece0 stored as bytes in memory (estimated size 209.0 B, free 1918.2 MiB) 15:45:49.193 INFO BlockManagerInfo - Added broadcast_1980_piece0 in memory on localhost:43633 (size: 209.0 B, free: 1919.6 MiB) 15:45:49.193 INFO SparkContext - Created broadcast 1980 from broadcast at MarkDuplicatesSparkUtils.java:127 15:45:49.211 INFO SparkContext - Starting job: collectAsMap at MarkDuplicatesSparkUtils.java:542 15:45:49.212 INFO DAGScheduler - Registering RDD 5459 (flatMapToPair at MarkDuplicatesSparkUtils.java:130) as input to shuffle 303 15:45:49.212 INFO DAGScheduler - Registering RDD 5463 (mapToPair at MarkDuplicatesSpark.java:217) as input to shuffle 302 15:45:49.212 INFO DAGScheduler - Registering RDD 5467 (mapToPair at MarkDuplicatesSparkUtils.java:497) as input to shuffle 301 15:45:49.212 INFO DAGScheduler - Got job 560 (collectAsMap at MarkDuplicatesSparkUtils.java:542) with 1 output partitions 15:45:49.212 INFO DAGScheduler - Final stage: ResultStage 1224 (collectAsMap at MarkDuplicatesSparkUtils.java:542) 15:45:49.212 INFO DAGScheduler - Parents of final stage: List(ShuffleMapStage 1223) 15:45:49.212 INFO DAGScheduler - Missing parents: List(ShuffleMapStage 1223) 15:45:49.213 INFO DAGScheduler - Submitting ShuffleMapStage 1221 (MapPartitionsRDD[5459] at flatMapToPair at MarkDuplicatesSparkUtils.java:130), which has no missing parents 15:45:49.214 INFO MemoryStore - Block broadcast_1981 stored as values in memory (estimated size 57.6 KiB, free 1918.1 MiB) 15:45:49.214 INFO MemoryStore - Block broadcast_1981_piece0 stored as bytes in memory (estimated size 19.8 KiB, free 1918.1 MiB) 15:45:49.215 INFO BlockManagerInfo - Added broadcast_1981_piece0 in memory on localhost:43633 (size: 19.8 KiB, free: 1919.6 MiB) 15:45:49.215 INFO SparkContext - Created broadcast 1981 from broadcast at DAGScheduler.scala:1580 15:45:49.215 INFO DAGScheduler - Submitting 1 missing tasks from ShuffleMapStage 1221 (MapPartitionsRDD[5459] at flatMapToPair at MarkDuplicatesSparkUtils.java:130) (first 15 tasks are for partitions Vector(0)) 15:45:49.215 INFO TaskSchedulerImpl - Adding task set 1221.0 with 1 tasks resource profile 0 15:45:49.215 INFO TaskSetManager - Starting task 0.0 in stage 1221.0 (TID 1064) (localhost, executor driver, partition 0, NODE_LOCAL, 9890 bytes) 15:45:49.216 INFO Executor - Running task 0.0 in stage 1221.0 (TID 1064) 15:45:49.218 INFO ShuffleBlockFetcherIterator - Getting 1 (16.3 KiB) non-empty blocks including 1 (16.3 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks 15:45:49.218 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms 15:45:49.224 INFO Executor - Finished task 0.0 in stage 1221.0 (TID 1064). 1922 bytes result sent to driver 15:45:49.224 INFO TaskSetManager - Finished task 0.0 in stage 1221.0 (TID 1064) in 9 ms on localhost (executor driver) (1/1) 15:45:49.224 INFO TaskSchedulerImpl - Removed TaskSet 1221.0, whose tasks have all completed, from pool 15:45:49.224 INFO DAGScheduler - ShuffleMapStage 1221 (flatMapToPair at MarkDuplicatesSparkUtils.java:130) finished in 0.011 s 15:45:49.224 INFO DAGScheduler - looking for newly runnable stages 15:45:49.224 INFO DAGScheduler - running: HashSet() 15:45:49.224 INFO DAGScheduler - waiting: HashSet(ShuffleMapStage 1222, ShuffleMapStage 1223, ResultStage 1224) 15:45:49.224 INFO DAGScheduler - failed: HashSet() 15:45:49.224 INFO DAGScheduler - Submitting ShuffleMapStage 1222 (MapPartitionsRDD[5463] at mapToPair at MarkDuplicatesSpark.java:217), which has no missing parents 15:45:49.226 INFO MemoryStore - Block broadcast_1982 stored as values in memory (estimated size 61.7 KiB, free 1918.1 MiB) 15:45:49.226 INFO MemoryStore - Block broadcast_1982_piece0 stored as bytes in memory (estimated size 22.0 KiB, free 1918.0 MiB) 15:45:49.226 INFO BlockManagerInfo - Added broadcast_1982_piece0 in memory on localhost:43633 (size: 22.0 KiB, free: 1919.6 MiB) 15:45:49.226 INFO SparkContext - Created broadcast 1982 from broadcast at DAGScheduler.scala:1580 15:45:49.227 INFO DAGScheduler - Submitting 1 missing tasks from ShuffleMapStage 1222 (MapPartitionsRDD[5463] at mapToPair at MarkDuplicatesSpark.java:217) (first 15 tasks are for partitions Vector(0)) 15:45:49.227 INFO TaskSchedulerImpl - Adding task set 1222.0 with 1 tasks resource profile 0 15:45:49.227 INFO TaskSetManager - Starting task 0.0 in stage 1222.0 (TID 1065) (localhost, executor driver, partition 0, NODE_LOCAL, 9365 bytes) 15:45:49.227 INFO Executor - Running task 0.0 in stage 1222.0 (TID 1065) 15:45:49.229 INFO ShuffleBlockFetcherIterator - Getting 1 (2.0 KiB) non-empty blocks including 1 (2.0 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks 15:45:49.230 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms 15:45:49.237 INFO Executor - Finished task 0.0 in stage 1222.0 (TID 1065). 1922 bytes result sent to driver 15:45:49.237 INFO TaskSetManager - Finished task 0.0 in stage 1222.0 (TID 1065) in 10 ms on localhost (executor driver) (1/1) 15:45:49.237 INFO DAGScheduler - ShuffleMapStage 1222 (mapToPair at MarkDuplicatesSpark.java:217) finished in 0.013 s 15:45:49.237 INFO DAGScheduler - looking for newly runnable stages 15:45:49.237 INFO DAGScheduler - running: HashSet() 15:45:49.238 INFO DAGScheduler - waiting: HashSet(ShuffleMapStage 1223, ResultStage 1224) 15:45:49.238 INFO DAGScheduler - failed: HashSet() 15:45:49.238 INFO DAGScheduler - Submitting ShuffleMapStage 1223 (MapPartitionsRDD[5467] at mapToPair at MarkDuplicatesSparkUtils.java:497), which has no missing parents 15:45:49.238 INFO TaskSchedulerImpl - Removed TaskSet 1222.0, whose tasks have all completed, from pool 15:45:49.239 INFO MemoryStore - Block broadcast_1983 stored as values in memory (estimated size 52.4 KiB, free 1918.0 MiB) 15:45:49.239 INFO MemoryStore - Block broadcast_1983_piece0 stored as bytes in memory (estimated size 17.3 KiB, free 1918.0 MiB) 15:45:49.240 INFO BlockManagerInfo - Added broadcast_1983_piece0 in memory on localhost:43633 (size: 17.3 KiB, free: 1919.5 MiB) 15:45:49.240 INFO SparkContext - Created broadcast 1983 from broadcast at DAGScheduler.scala:1580 15:45:49.240 INFO DAGScheduler - Submitting 1 missing tasks from ShuffleMapStage 1223 (MapPartitionsRDD[5467] at mapToPair at MarkDuplicatesSparkUtils.java:497) (first 15 tasks are for partitions Vector(0)) 15:45:49.240 INFO TaskSchedulerImpl - Adding task set 1223.0 with 1 tasks resource profile 0 15:45:49.240 INFO TaskSetManager - Starting task 0.0 in stage 1223.0 (TID 1066) (localhost, executor driver, partition 0, NODE_LOCAL, 9948 bytes) 15:45:49.240 INFO Executor - Running task 0.0 in stage 1223.0 (TID 1066) 15:45:49.242 INFO ShuffleBlockFetcherIterator - Getting 1 (16.3 KiB) non-empty blocks including 1 (16.3 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks 15:45:49.242 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms 15:45:49.245 INFO ShuffleBlockFetcherIterator - Getting 1 (2.7 KiB) non-empty blocks including 1 (2.7 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks 15:45:49.245 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms 15:45:49.246 INFO Executor - Finished task 0.0 in stage 1223.0 (TID 1066). 1922 bytes result sent to driver 15:45:49.246 INFO TaskSetManager - Finished task 0.0 in stage 1223.0 (TID 1066) in 6 ms on localhost (executor driver) (1/1) 15:45:49.247 INFO TaskSchedulerImpl - Removed TaskSet 1223.0, whose tasks have all completed, from pool 15:45:49.247 INFO DAGScheduler - ShuffleMapStage 1223 (mapToPair at MarkDuplicatesSparkUtils.java:497) finished in 0.009 s 15:45:49.247 INFO DAGScheduler - looking for newly runnable stages 15:45:49.247 INFO DAGScheduler - running: HashSet() 15:45:49.247 INFO DAGScheduler - waiting: HashSet(ResultStage 1224) 15:45:49.247 INFO DAGScheduler - failed: HashSet() 15:45:49.247 INFO DAGScheduler - Submitting ResultStage 1224 (MapPartitionsRDD[5469] at mapValues at MarkDuplicatesSparkUtils.java:519), which has no missing parents 15:45:49.248 INFO MemoryStore - Block broadcast_1984 stored as values in memory (estimated size 53.6 KiB, free 1917.9 MiB) 15:45:49.248 INFO MemoryStore - Block broadcast_1984_piece0 stored as bytes in memory (estimated size 18.0 KiB, free 1917.9 MiB) 15:45:49.248 INFO BlockManagerInfo - Added broadcast_1984_piece0 in memory on localhost:43633 (size: 18.0 KiB, free: 1919.5 MiB) 15:45:49.248 INFO SparkContext - Created broadcast 1984 from broadcast at DAGScheduler.scala:1580 15:45:49.249 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 1224 (MapPartitionsRDD[5469] at mapValues at MarkDuplicatesSparkUtils.java:519) (first 15 tasks are for partitions Vector(0)) 15:45:49.249 INFO TaskSchedulerImpl - Adding task set 1224.0 with 1 tasks resource profile 0 15:45:49.249 INFO TaskSetManager - Starting task 0.0 in stage 1224.0 (TID 1067) (localhost, executor driver, partition 0, NODE_LOCAL, 9376 bytes) 15:45:49.249 INFO Executor - Running task 0.0 in stage 1224.0 (TID 1067) 15:45:49.250 INFO ShuffleBlockFetcherIterator - Getting 1 (251.0 B) non-empty blocks including 1 (251.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks 15:45:49.250 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms 15:45:49.252 INFO Executor - Finished task 0.0 in stage 1224.0 (TID 1067). 2371 bytes result sent to driver 15:45:49.252 INFO TaskSetManager - Finished task 0.0 in stage 1224.0 (TID 1067) in 3 ms on localhost (executor driver) (1/1) 15:45:49.252 INFO TaskSchedulerImpl - Removed TaskSet 1224.0, whose tasks have all completed, from pool 15:45:49.252 INFO DAGScheduler - ResultStage 1224 (collectAsMap at MarkDuplicatesSparkUtils.java:542) finished in 0.005 s 15:45:49.252 INFO DAGScheduler - Job 560 is finished. Cancelling potential speculative or zombie tasks for this job 15:45:49.252 INFO TaskSchedulerImpl - Killing all running tasks in stage 1224: Stage finished 15:45:49.252 INFO DAGScheduler - Job 560 finished: collectAsMap at MarkDuplicatesSparkUtils.java:542, took 0.040571 s 15:45:49.264 INFO MemoryStore - Block broadcast_1985 stored as values in memory (estimated size 179.4 KiB, free 1917.7 MiB) 15:45:49.264 INFO MemoryStore - Block broadcast_1985_piece0 stored as bytes in memory (estimated size 7.9 KiB, free 1917.7 MiB) 15:45:49.265 INFO BlockManagerInfo - Added broadcast_1985_piece0 in memory on localhost:43633 (size: 7.9 KiB, free: 1919.5 MiB) 15:45:49.265 INFO SparkContext - Created broadcast 1985 from broadcast at ReadsSparkSink.java:133 15:45:49.268 INFO HadoopMapRedCommitProtocol - Using output committer class org.apache.hadoop.mapred.FileOutputCommitter 15:45:49.268 INFO FileOutputCommitter - File Output Committer Algorithm version is 1 15:45:49.268 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false 15:45:49.284 INFO SparkContext - Starting job: runJob at SparkHadoopWriter.scala:83 15:45:49.285 INFO DAGScheduler - Registering RDD 5471 (mapToPair at SparkUtils.java:161) as input to shuffle 304 15:45:49.285 INFO DAGScheduler - Got job 561 (runJob at SparkHadoopWriter.scala:83) with 1 output partitions 15:45:49.285 INFO DAGScheduler - Final stage: ResultStage 1229 (runJob at SparkHadoopWriter.scala:83) 15:45:49.285 INFO DAGScheduler - Parents of final stage: List(ShuffleMapStage 1228) 15:45:49.285 INFO DAGScheduler - Missing parents: List(ShuffleMapStage 1228) 15:45:49.285 INFO DAGScheduler - Submitting ShuffleMapStage 1228 (MapPartitionsRDD[5471] at mapToPair at SparkUtils.java:161), which has no missing parents 15:45:49.286 INFO MemoryStore - Block broadcast_1986 stored as values in memory (estimated size 51.5 KiB, free 1917.7 MiB) 15:45:49.287 INFO MemoryStore - Block broadcast_1986_piece0 stored as bytes in memory (estimated size 16.8 KiB, free 1917.6 MiB) 15:45:49.287 INFO BlockManagerInfo - Added broadcast_1986_piece0 in memory on localhost:43633 (size: 16.8 KiB, free: 1919.5 MiB) 15:45:49.287 INFO SparkContext - Created broadcast 1986 from broadcast at DAGScheduler.scala:1580 15:45:49.287 INFO DAGScheduler - Submitting 1 missing tasks from ShuffleMapStage 1228 (MapPartitionsRDD[5471] at mapToPair at SparkUtils.java:161) (first 15 tasks are for partitions Vector(0)) 15:45:49.287 INFO TaskSchedulerImpl - Adding task set 1228.0 with 1 tasks resource profile 0 15:45:49.288 INFO TaskSetManager - Starting task 0.0 in stage 1228.0 (TID 1068) (localhost, executor driver, partition 0, NODE_LOCAL, 9948 bytes) 15:45:49.288 INFO Executor - Running task 0.0 in stage 1228.0 (TID 1068) 15:45:49.289 INFO ShuffleBlockFetcherIterator - Getting 1 (16.3 KiB) non-empty blocks including 1 (16.3 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks 15:45:49.289 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms 15:45:49.292 INFO ShuffleBlockFetcherIterator - Getting 1 (2.7 KiB) non-empty blocks including 1 (2.7 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks 15:45:49.292 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms 15:45:49.293 INFO Executor - Finished task 0.0 in stage 1228.0 (TID 1068). 1922 bytes result sent to driver 15:45:49.294 INFO TaskSetManager - Finished task 0.0 in stage 1228.0 (TID 1068) in 7 ms on localhost (executor driver) (1/1) 15:45:49.294 INFO TaskSchedulerImpl - Removed TaskSet 1228.0, whose tasks have all completed, from pool 15:45:49.294 INFO DAGScheduler - ShuffleMapStage 1228 (mapToPair at SparkUtils.java:161) finished in 0.008 s 15:45:49.294 INFO DAGScheduler - looking for newly runnable stages 15:45:49.294 INFO DAGScheduler - running: HashSet() 15:45:49.294 INFO DAGScheduler - waiting: HashSet(ResultStage 1229) 15:45:49.294 INFO DAGScheduler - failed: HashSet() 15:45:49.294 INFO DAGScheduler - Submitting ResultStage 1229 (MapPartitionsRDD[5477] at saveAsTextFile at SamSink.java:65), which has no missing parents 15:45:49.300 INFO MemoryStore - Block broadcast_1987 stored as values in memory (estimated size 196.7 KiB, free 1917.5 MiB) 15:45:49.301 INFO MemoryStore - Block broadcast_1987_piece0 stored as bytes in memory (estimated size 80.9 KiB, free 1917.4 MiB) 15:45:49.301 INFO BlockManagerInfo - Added broadcast_1987_piece0 in memory on localhost:43633 (size: 80.9 KiB, free: 1919.4 MiB) 15:45:49.302 INFO SparkContext - Created broadcast 1987 from broadcast at DAGScheduler.scala:1580 15:45:49.302 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 1229 (MapPartitionsRDD[5477] at saveAsTextFile at SamSink.java:65) (first 15 tasks are for partitions Vector(0)) 15:45:49.302 INFO TaskSchedulerImpl - Adding task set 1229.0 with 1 tasks resource profile 0 15:45:49.302 INFO TaskSetManager - Starting task 0.0 in stage 1229.0 (TID 1069) (localhost, executor driver, partition 0, NODE_LOCAL, 9376 bytes) 15:45:49.302 INFO Executor - Running task 0.0 in stage 1229.0 (TID 1069) 15:45:49.306 INFO ShuffleBlockFetcherIterator - Getting 1 (14.8 KiB) non-empty blocks including 1 (14.8 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks 15:45:49.306 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms 15:45:49.308 INFO HadoopMapRedCommitProtocol - Using output committer class org.apache.hadoop.mapred.FileOutputCommitter 15:45:49.308 INFO FileOutputCommitter - File Output Committer Algorithm version is 1 15:45:49.308 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false 15:45:49.319 INFO FileOutputCommitter - Saved output of task 'attempt_2025052715454931453362556181564_5477_m_000000_0' to file:/tmp/MarkDups11835526939909527871/MarkDups.sam.parts/_temporary/0/task_2025052715454931453362556181564_5477_m_000000 15:45:49.319 INFO SparkHadoopMapRedUtil - attempt_2025052715454931453362556181564_5477_m_000000_0: Committed. Elapsed time: 0 ms. 15:45:49.319 INFO Executor - Finished task 0.0 in stage 1229.0 (TID 1069). 1858 bytes result sent to driver 15:45:49.319 INFO TaskSetManager - Finished task 0.0 in stage 1229.0 (TID 1069) in 17 ms on localhost (executor driver) (1/1) 15:45:49.319 INFO DAGScheduler - ResultStage 1229 (runJob at SparkHadoopWriter.scala:83) finished in 0.025 s 15:45:49.319 INFO TaskSchedulerImpl - Removed TaskSet 1229.0, whose tasks have all completed, from pool 15:45:49.320 INFO DAGScheduler - Job 561 is finished. Cancelling potential speculative or zombie tasks for this job 15:45:49.320 INFO TaskSchedulerImpl - Killing all running tasks in stage 1229: Stage finished 15:45:49.320 INFO DAGScheduler - Job 561 finished: runJob at SparkHadoopWriter.scala:83, took 0.035248 s 15:45:49.320 INFO SparkHadoopWriter - Start to commit write Job job_2025052715454931453362556181564_5477. 15:45:49.324 INFO SparkHadoopWriter - Write Job job_2025052715454931453362556181564_5477 committed. Elapsed time: 4 ms. 15:45:49.331 INFO HadoopFileSystemWrapper - Concatenating 2 parts to /tmp/MarkDups11835526939909527871/MarkDups.sam 15:45:49.335 INFO HadoopFileSystemWrapper - Concatenating to /tmp/MarkDups11835526939909527871/MarkDups.sam done 15:45:49.335 INFO MarkDuplicatesSpark - Shutting down engine [May 27, 2025 at 3:45:49 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.01 minutes. Runtime.totalMemory()=1426063360 15:45:49.341 INFO MarkDuplicatesSpark - ------------------------------------------------------------ 15:45:49.341 INFO MarkDuplicatesSpark - org.broadinstitute.hellbender.tools.spark.transforms.markduplicates vUnavailable 15:45:49.341 INFO MarkDuplicatesSpark - For support and documentation go to https://software.broadinstitute.org/gatk/ 15:45:49.341 INFO MarkDuplicatesSpark - Executing as runner@pkrvmf6wy0o8zjz on Linux v6.11.0-1014-azure amd64 15:45:49.341 INFO MarkDuplicatesSpark - Java runtime: OpenJDK 64-Bit Server VM v17.0.6+10 15:45:49.341 INFO MarkDuplicatesSpark - Start Date/Time: May 27, 2025 at 3:45:49 PM UTC 15:45:49.341 INFO MarkDuplicatesSpark - ------------------------------------------------------------ 15:45:49.341 INFO MarkDuplicatesSpark - ------------------------------------------------------------ 15:45:49.341 INFO MarkDuplicatesSpark - HTSJDK Defaults.COMPRESSION_LEVEL : 2 15:45:49.341 INFO MarkDuplicatesSpark - HTSJDK Defaults.USE_ASYNC_IO_READ_FOR_SAMTOOLS : false 15:45:49.341 INFO MarkDuplicatesSpark - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_SAMTOOLS : true 15:45:49.341 INFO MarkDuplicatesSpark - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_TRIBBLE : false 15:45:49.341 INFO MarkDuplicatesSpark - Deflater: IntelDeflater 15:45:49.341 INFO MarkDuplicatesSpark - Inflater: IntelInflater 15:45:49.341 INFO MarkDuplicatesSpark - GCS max retries/reopens: 20 15:45:49.341 INFO MarkDuplicatesSpark - Requester pays: disabled 15:45:49.341 INFO MarkDuplicatesSpark - Initializing engine 15:45:49.341 INFO MarkDuplicatesSpark - Done initializing engine 15:45:49.341 INFO MarkDuplicatesSpark - Spark verbosity set to INFO (see --spark-verbosity argument) 15:45:49.343 INFO MemoryStore - Block broadcast_1988 stored as values in memory (estimated size 306.3 KiB, free 1917.1 MiB) 15:45:49.350 INFO MemoryStore - Block broadcast_1988_piece0 stored as bytes in memory (estimated size 64.4 KiB, free 1917.0 MiB) 15:45:49.350 INFO BlockManagerInfo - Added broadcast_1988_piece0 in memory on localhost:43633 (size: 64.4 KiB, free: 1919.4 MiB) 15:45:49.350 INFO SparkContext - Created broadcast 1988 from newAPIHadoopFile at PathSplitSource.java:96 15:45:49.370 INFO MemoryStore - Block broadcast_1989 stored as values in memory (estimated size 306.3 KiB, free 1916.7 MiB) 15:45:49.376 INFO MemoryStore - Block broadcast_1989_piece0 stored as bytes in memory (estimated size 64.4 KiB, free 1916.6 MiB) 15:45:49.376 INFO BlockManagerInfo - Added broadcast_1989_piece0 in memory on localhost:43633 (size: 64.4 KiB, free: 1919.3 MiB) 15:45:49.376 INFO SparkContext - Created broadcast 1989 from newAPIHadoopFile at PathSplitSource.java:96 15:45:49.408 INFO SparkContext - Starting job: collect at SparkUtils.java:205 15:45:49.408 INFO FileInputFormat - Total input files to process : 1 15:45:49.409 INFO DAGScheduler - Registering RDD 5491 (mapToPair at SparkUtils.java:161) as input to shuffle 305 15:45:49.409 INFO DAGScheduler - Got job 562 (collect at SparkUtils.java:205) with 1 output partitions 15:45:49.409 INFO DAGScheduler - Final stage: ResultStage 1231 (collect at SparkUtils.java:205) 15:45:49.409 INFO DAGScheduler - Parents of final stage: List(ShuffleMapStage 1230) 15:45:49.409 INFO DAGScheduler - Missing parents: List(ShuffleMapStage 1230) 15:45:49.409 INFO DAGScheduler - Submitting ShuffleMapStage 1230 (MapPartitionsRDD[5491] at mapToPair at SparkUtils.java:161), which has no missing parents 15:45:49.425 INFO MemoryStore - Block broadcast_1990 stored as values in memory (estimated size 457.4 KiB, free 1916.2 MiB) 15:45:49.428 INFO MemoryStore - Block broadcast_1990_piece0 stored as bytes in memory (estimated size 201.1 KiB, free 1916.0 MiB) 15:45:49.428 INFO BlockManagerInfo - Added broadcast_1990_piece0 in memory on localhost:43633 (size: 201.1 KiB, free: 1919.1 MiB) 15:45:49.428 INFO SparkContext - Created broadcast 1990 from broadcast at DAGScheduler.scala:1580 15:45:49.428 INFO DAGScheduler - Submitting 1 missing tasks from ShuffleMapStage 1230 (MapPartitionsRDD[5491] at mapToPair at SparkUtils.java:161) (first 15 tasks are for partitions Vector(0)) 15:45:49.428 INFO TaskSchedulerImpl - Adding task set 1230.0 with 1 tasks resource profile 0 15:45:49.428 INFO TaskSetManager - Starting task 0.0 in stage 1230.0 (TID 1070) (localhost, executor driver, partition 0, PROCESS_LOCAL, 9738 bytes) 15:45:49.429 INFO Executor - Running task 0.0 in stage 1230.0 (TID 1070) 15:45:49.457 INFO NewHadoopRDD - Input split: file:/home/runner/work/gatk/gatk/src/test/resources/org/broadinstitute/hellbender/tools/walkers/MarkDuplicatesGATK/optical_dupes.bam:0+678 15:45:49.460 INFO Executor - Finished task 0.0 in stage 1230.0 (TID 1070). 1148 bytes result sent to driver 15:45:49.460 INFO TaskSetManager - Finished task 0.0 in stage 1230.0 (TID 1070) in 32 ms on localhost (executor driver) (1/1) 15:45:49.461 INFO TaskSchedulerImpl - Removed TaskSet 1230.0, whose tasks have all completed, from pool 15:45:49.461 INFO DAGScheduler - ShuffleMapStage 1230 (mapToPair at SparkUtils.java:161) finished in 0.052 s 15:45:49.461 INFO DAGScheduler - looking for newly runnable stages 15:45:49.461 INFO DAGScheduler - running: HashSet() 15:45:49.461 INFO DAGScheduler - waiting: HashSet(ResultStage 1231) 15:45:49.461 INFO DAGScheduler - failed: HashSet() 15:45:49.461 INFO DAGScheduler - Submitting ResultStage 1231 (MapPartitionsRDD[5494] at mapPartitions at SparkUtils.java:188), which has no missing parents 15:45:49.462 INFO MemoryStore - Block broadcast_1991 stored as values in memory (estimated size 7.4 KiB, free 1916.0 MiB) 15:45:49.462 INFO MemoryStore - Block broadcast_1991_piece0 stored as bytes in memory (estimated size 4.1 KiB, free 1916.0 MiB) 15:45:49.462 INFO BlockManagerInfo - Added broadcast_1991_piece0 in memory on localhost:43633 (size: 4.1 KiB, free: 1919.1 MiB) 15:45:49.462 INFO SparkContext - Created broadcast 1991 from broadcast at DAGScheduler.scala:1580 15:45:49.462 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 1231 (MapPartitionsRDD[5494] at mapPartitions at SparkUtils.java:188) (first 15 tasks are for partitions Vector(0)) 15:45:49.462 INFO TaskSchedulerImpl - Adding task set 1231.0 with 1 tasks resource profile 0 15:45:49.463 INFO TaskSetManager - Starting task 0.0 in stage 1231.0 (TID 1071) (localhost, executor driver, partition 0, NODE_LOCAL, 9376 bytes) 15:45:49.463 INFO Executor - Running task 0.0 in stage 1231.0 (TID 1071) 15:45:49.464 INFO ShuffleBlockFetcherIterator - Getting 1 (490.0 B) non-empty blocks including 1 (490.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks 15:45:49.464 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms 15:45:49.465 INFO Executor - Finished task 0.0 in stage 1231.0 (TID 1071). 2269 bytes result sent to driver 15:45:49.465 INFO TaskSetManager - Finished task 0.0 in stage 1231.0 (TID 1071) in 2 ms on localhost (executor driver) (1/1) 15:45:49.465 INFO TaskSchedulerImpl - Removed TaskSet 1231.0, whose tasks have all completed, from pool 15:45:49.465 INFO DAGScheduler - ResultStage 1231 (collect at SparkUtils.java:205) finished in 0.004 s 15:45:49.466 INFO DAGScheduler - Job 562 is finished. Cancelling potential speculative or zombie tasks for this job 15:45:49.466 INFO TaskSchedulerImpl - Killing all running tasks in stage 1231: Stage finished 15:45:49.466 INFO DAGScheduler - Job 562 finished: collect at SparkUtils.java:205, took 0.057709 s 15:45:49.470 INFO MemoryStore - Block broadcast_1992 stored as values in memory (estimated size 136.0 B, free 1916.0 MiB) 15:45:49.470 INFO MemoryStore - Block broadcast_1992_piece0 stored as bytes in memory (estimated size 24.0 B, free 1916.0 MiB) 15:45:49.470 INFO BlockManagerInfo - Added broadcast_1992_piece0 in memory on localhost:43633 (size: 24.0 B, free: 1919.1 MiB) 15:45:49.470 INFO SparkContext - Created broadcast 1992 from broadcast at MarkDuplicatesSparkUtils.java:126 15:45:49.470 INFO MemoryStore - Block broadcast_1993 stored as values in memory (estimated size 136.0 B, free 1916.0 MiB) 15:45:49.471 INFO MemoryStore - Block broadcast_1993_piece0 stored as bytes in memory (estimated size 21.0 B, free 1916.0 MiB) 15:45:49.471 INFO BlockManagerInfo - Added broadcast_1993_piece0 in memory on localhost:43633 (size: 21.0 B, free: 1919.1 MiB) 15:45:49.471 INFO SparkContext - Created broadcast 1993 from broadcast at MarkDuplicatesSparkUtils.java:127 15:45:49.482 INFO SparkContext - Starting job: collectAsMap at MarkDuplicatesSparkUtils.java:542 15:45:49.483 INFO DAGScheduler - Registering RDD 5503 (flatMapToPair at MarkDuplicatesSparkUtils.java:130) as input to shuffle 308 15:45:49.483 INFO DAGScheduler - Registering RDD 5507 (mapToPair at MarkDuplicatesSpark.java:217) as input to shuffle 307 15:45:49.483 INFO DAGScheduler - Registering RDD 5511 (mapToPair at MarkDuplicatesSparkUtils.java:497) as input to shuffle 306 15:45:49.483 INFO DAGScheduler - Got job 563 (collectAsMap at MarkDuplicatesSparkUtils.java:542) with 1 output partitions 15:45:49.483 INFO DAGScheduler - Final stage: ResultStage 1236 (collectAsMap at MarkDuplicatesSparkUtils.java:542) 15:45:49.483 INFO DAGScheduler - Parents of final stage: List(ShuffleMapStage 1235) 15:45:49.483 INFO DAGScheduler - Missing parents: List(ShuffleMapStage 1235) 15:45:49.483 INFO DAGScheduler - Submitting ShuffleMapStage 1233 (MapPartitionsRDD[5503] at flatMapToPair at MarkDuplicatesSparkUtils.java:130), which has no missing parents 15:45:49.484 INFO MemoryStore - Block broadcast_1994 stored as values in memory (estimated size 21.1 KiB, free 1916.0 MiB) 15:45:49.485 INFO MemoryStore - Block broadcast_1994_piece0 stored as bytes in memory (estimated size 10.9 KiB, free 1916.0 MiB) 15:45:49.485 INFO BlockManagerInfo - Added broadcast_1994_piece0 in memory on localhost:43633 (size: 10.9 KiB, free: 1919.1 MiB) 15:45:49.485 INFO SparkContext - Created broadcast 1994 from broadcast at DAGScheduler.scala:1580 15:45:49.485 INFO DAGScheduler - Submitting 1 missing tasks from ShuffleMapStage 1233 (MapPartitionsRDD[5503] at flatMapToPair at MarkDuplicatesSparkUtils.java:130) (first 15 tasks are for partitions Vector(0)) 15:45:49.485 INFO TaskSchedulerImpl - Adding task set 1233.0 with 1 tasks resource profile 0 15:45:49.485 INFO TaskSetManager - Starting task 0.0 in stage 1233.0 (TID 1072) (localhost, executor driver, partition 0, NODE_LOCAL, 9890 bytes) 15:45:49.486 INFO Executor - Running task 0.0 in stage 1233.0 (TID 1072) 15:45:49.487 INFO ShuffleBlockFetcherIterator - Getting 1 (490.0 B) non-empty blocks including 1 (490.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks 15:45:49.487 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms 15:45:49.489 INFO Executor - Finished task 0.0 in stage 1233.0 (TID 1072). 1922 bytes result sent to driver 15:45:49.489 INFO TaskSetManager - Finished task 0.0 in stage 1233.0 (TID 1072) in 4 ms on localhost (executor driver) (1/1) 15:45:49.489 INFO TaskSchedulerImpl - Removed TaskSet 1233.0, whose tasks have all completed, from pool 15:45:49.489 INFO DAGScheduler - ShuffleMapStage 1233 (flatMapToPair at MarkDuplicatesSparkUtils.java:130) finished in 0.005 s 15:45:49.490 INFO DAGScheduler - looking for newly runnable stages 15:45:49.490 INFO DAGScheduler - running: HashSet() 15:45:49.490 INFO DAGScheduler - waiting: HashSet(ShuffleMapStage 1234, ShuffleMapStage 1235, ResultStage 1236) 15:45:49.490 INFO DAGScheduler - failed: HashSet() 15:45:49.490 INFO DAGScheduler - Submitting ShuffleMapStage 1234 (MapPartitionsRDD[5507] at mapToPair at MarkDuplicatesSpark.java:217), which has no missing parents 15:45:49.490 INFO MemoryStore - Block broadcast_1995 stored as values in memory (estimated size 25.2 KiB, free 1915.9 MiB) 15:45:49.491 INFO MemoryStore - Block broadcast_1995_piece0 stored as bytes in memory (estimated size 13.1 KiB, free 1915.9 MiB) 15:45:49.491 INFO BlockManagerInfo - Added broadcast_1995_piece0 in memory on localhost:43633 (size: 13.1 KiB, free: 1919.1 MiB) 15:45:49.491 INFO SparkContext - Created broadcast 1995 from broadcast at DAGScheduler.scala:1580 15:45:49.491 INFO DAGScheduler - Submitting 1 missing tasks from ShuffleMapStage 1234 (MapPartitionsRDD[5507] at mapToPair at MarkDuplicatesSpark.java:217) (first 15 tasks are for partitions Vector(0)) 15:45:49.491 INFO TaskSchedulerImpl - Adding task set 1234.0 with 1 tasks resource profile 0 15:45:49.492 INFO TaskSetManager - Starting task 0.0 in stage 1234.0 (TID 1073) (localhost, executor driver, partition 0, NODE_LOCAL, 9365 bytes) 15:45:49.492 INFO Executor - Running task 0.0 in stage 1234.0 (TID 1073) 15:45:49.493 INFO ShuffleBlockFetcherIterator - Getting 1 (106.0 B) non-empty blocks including 1 (106.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks 15:45:49.493 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms 15:45:49.496 INFO Executor - Finished task 0.0 in stage 1234.0 (TID 1073). 1879 bytes result sent to driver 15:45:49.496 INFO TaskSetManager - Finished task 0.0 in stage 1234.0 (TID 1073) in 5 ms on localhost (executor driver) (1/1) 15:45:49.496 INFO TaskSchedulerImpl - Removed TaskSet 1234.0, whose tasks have all completed, from pool 15:45:49.496 INFO DAGScheduler - ShuffleMapStage 1234 (mapToPair at MarkDuplicatesSpark.java:217) finished in 0.006 s 15:45:49.496 INFO DAGScheduler - looking for newly runnable stages 15:45:49.496 INFO DAGScheduler - running: HashSet() 15:45:49.496 INFO DAGScheduler - waiting: HashSet(ShuffleMapStage 1235, ResultStage 1236) 15:45:49.496 INFO DAGScheduler - failed: HashSet() 15:45:49.497 INFO DAGScheduler - Submitting ShuffleMapStage 1235 (MapPartitionsRDD[5511] at mapToPair at MarkDuplicatesSparkUtils.java:497), which has no missing parents 15:45:49.497 INFO MemoryStore - Block broadcast_1996 stored as values in memory (estimated size 15.9 KiB, free 1915.9 MiB) 15:45:49.497 INFO MemoryStore - Block broadcast_1996_piece0 stored as bytes in memory (estimated size 8.3 KiB, free 1915.9 MiB) 15:45:49.498 INFO BlockManagerInfo - Added broadcast_1996_piece0 in memory on localhost:43633 (size: 8.3 KiB, free: 1919.1 MiB) 15:45:49.498 INFO SparkContext - Created broadcast 1996 from broadcast at DAGScheduler.scala:1580 15:45:49.498 INFO DAGScheduler - Submitting 1 missing tasks from ShuffleMapStage 1235 (MapPartitionsRDD[5511] at mapToPair at MarkDuplicatesSparkUtils.java:497) (first 15 tasks are for partitions Vector(0)) 15:45:49.498 INFO TaskSchedulerImpl - Adding task set 1235.0 with 1 tasks resource profile 0 15:45:49.498 INFO TaskSetManager - Starting task 0.0 in stage 1235.0 (TID 1074) (localhost, executor driver, partition 0, NODE_LOCAL, 9948 bytes) 15:45:49.498 INFO Executor - Running task 0.0 in stage 1235.0 (TID 1074) 15:45:49.499 INFO ShuffleBlockFetcherIterator - Getting 1 (490.0 B) non-empty blocks including 1 (490.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks 15:45:49.500 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms 15:45:49.501 INFO ShuffleBlockFetcherIterator - Getting 1 (60.0 B) non-empty blocks including 1 (60.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks 15:45:49.501 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms 15:45:49.502 INFO Executor - Finished task 0.0 in stage 1235.0 (TID 1074). 1879 bytes result sent to driver 15:45:49.502 INFO TaskSetManager - Finished task 0.0 in stage 1235.0 (TID 1074) in 4 ms on localhost (executor driver) (1/1) 15:45:49.502 INFO TaskSchedulerImpl - Removed TaskSet 1235.0, whose tasks have all completed, from pool 15:45:49.502 INFO DAGScheduler - ShuffleMapStage 1235 (mapToPair at MarkDuplicatesSparkUtils.java:497) finished in 0.005 s 15:45:49.502 INFO DAGScheduler - looking for newly runnable stages 15:45:49.502 INFO DAGScheduler - running: HashSet() 15:45:49.502 INFO DAGScheduler - waiting: HashSet(ResultStage 1236) 15:45:49.502 INFO DAGScheduler - failed: HashSet() 15:45:49.502 INFO DAGScheduler - Submitting ResultStage 1236 (MapPartitionsRDD[5513] at mapValues at MarkDuplicatesSparkUtils.java:519), which has no missing parents 15:45:49.503 INFO MemoryStore - Block broadcast_1997 stored as values in memory (estimated size 17.1 KiB, free 1915.9 MiB) 15:45:49.503 INFO MemoryStore - Block broadcast_1997_piece0 stored as bytes in memory (estimated size 9.0 KiB, free 1915.9 MiB) 15:45:49.503 INFO BlockManagerInfo - Added broadcast_1997_piece0 in memory on localhost:43633 (size: 9.0 KiB, free: 1919.0 MiB) 15:45:49.503 INFO SparkContext - Created broadcast 1997 from broadcast at DAGScheduler.scala:1580 15:45:49.504 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 1236 (MapPartitionsRDD[5513] at mapValues at MarkDuplicatesSparkUtils.java:519) (first 15 tasks are for partitions Vector(0)) 15:45:49.504 INFO TaskSchedulerImpl - Adding task set 1236.0 with 1 tasks resource profile 0 15:45:49.504 INFO TaskSetManager - Starting task 0.0 in stage 1236.0 (TID 1075) (localhost, executor driver, partition 0, NODE_LOCAL, 9376 bytes) 15:45:49.504 INFO Executor - Running task 0.0 in stage 1236.0 (TID 1075) 15:45:49.505 INFO ShuffleBlockFetcherIterator - Getting 1 (129.0 B) non-empty blocks including 1 (129.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks 15:45:49.505 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms 15:45:49.507 INFO Executor - Finished task 0.0 in stage 1236.0 (TID 1075). 1930 bytes result sent to driver 15:45:49.507 INFO TaskSetManager - Finished task 0.0 in stage 1236.0 (TID 1075) in 3 ms on localhost (executor driver) (1/1) 15:45:49.507 INFO TaskSchedulerImpl - Removed TaskSet 1236.0, whose tasks have all completed, from pool 15:45:49.507 INFO DAGScheduler - ResultStage 1236 (collectAsMap at MarkDuplicatesSparkUtils.java:542) finished in 0.005 s 15:45:49.507 INFO DAGScheduler - Job 563 is finished. Cancelling potential speculative or zombie tasks for this job 15:45:49.507 INFO TaskSchedulerImpl - Killing all running tasks in stage 1236: Stage finished 15:45:49.508 INFO DAGScheduler - Job 563 finished: collectAsMap at MarkDuplicatesSparkUtils.java:542, took 0.025263 s 15:45:49.518 INFO MemoryStore - Block broadcast_1998 stored as values in memory (estimated size 5.6 KiB, free 1915.9 MiB) 15:45:49.518 INFO MemoryStore - Block broadcast_1998_piece0 stored as bytes in memory (estimated size 500.0 B, free 1915.9 MiB) 15:45:49.518 INFO BlockManagerInfo - Added broadcast_1998_piece0 in memory on localhost:43633 (size: 500.0 B, free: 1919.0 MiB) 15:45:49.518 INFO SparkContext - Created broadcast 1998 from broadcast at ReadsSparkSink.java:133 15:45:49.521 INFO HadoopMapRedCommitProtocol - Using output committer class org.apache.hadoop.mapred.FileOutputCommitter 15:45:49.521 INFO FileOutputCommitter - File Output Committer Algorithm version is 1 15:45:49.521 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false 15:45:49.538 INFO SparkContext - Starting job: runJob at SparkHadoopWriter.scala:83 15:45:49.538 INFO DAGScheduler - Registering RDD 5515 (mapToPair at SparkUtils.java:161) as input to shuffle 309 15:45:49.539 INFO DAGScheduler - Got job 564 (runJob at SparkHadoopWriter.scala:83) with 1 output partitions 15:45:49.539 INFO DAGScheduler - Final stage: ResultStage 1241 (runJob at SparkHadoopWriter.scala:83) 15:45:49.539 INFO DAGScheduler - Parents of final stage: List(ShuffleMapStage 1240) 15:45:49.539 INFO DAGScheduler - Missing parents: List(ShuffleMapStage 1240) 15:45:49.539 INFO DAGScheduler - Submitting ShuffleMapStage 1240 (MapPartitionsRDD[5515] at mapToPair at SparkUtils.java:161), which has no missing parents 15:45:49.539 INFO MemoryStore - Block broadcast_1999 stored as values in memory (estimated size 15.0 KiB, free 1915.9 MiB) 15:45:49.540 INFO MemoryStore - Block broadcast_1999_piece0 stored as bytes in memory (estimated size 7.8 KiB, free 1915.8 MiB) 15:45:49.540 INFO BlockManagerInfo - Added broadcast_1999_piece0 in memory on localhost:43633 (size: 7.8 KiB, free: 1919.0 MiB) 15:45:49.540 INFO SparkContext - Created broadcast 1999 from broadcast at DAGScheduler.scala:1580 15:45:49.540 INFO DAGScheduler - Submitting 1 missing tasks from ShuffleMapStage 1240 (MapPartitionsRDD[5515] at mapToPair at SparkUtils.java:161) (first 15 tasks are for partitions Vector(0)) 15:45:49.540 INFO TaskSchedulerImpl - Adding task set 1240.0 with 1 tasks resource profile 0 15:45:49.541 INFO TaskSetManager - Starting task 0.0 in stage 1240.0 (TID 1076) (localhost, executor driver, partition 0, NODE_LOCAL, 9948 bytes) 15:45:49.541 INFO Executor - Running task 0.0 in stage 1240.0 (TID 1076) 15:45:49.542 INFO ShuffleBlockFetcherIterator - Getting 1 (490.0 B) non-empty blocks including 1 (490.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks 15:45:49.542 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms 15:45:49.544 INFO ShuffleBlockFetcherIterator - Getting 1 (60.0 B) non-empty blocks including 1 (60.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks 15:45:49.544 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms 15:45:49.545 INFO Executor - Finished task 0.0 in stage 1240.0 (TID 1076). 1879 bytes result sent to driver 15:45:49.545 INFO TaskSetManager - Finished task 0.0 in stage 1240.0 (TID 1076) in 5 ms on localhost (executor driver) (1/1) 15:45:49.545 INFO TaskSchedulerImpl - Removed TaskSet 1240.0, whose tasks have all completed, from pool 15:45:49.545 INFO DAGScheduler - ShuffleMapStage 1240 (mapToPair at SparkUtils.java:161) finished in 0.006 s 15:45:49.545 INFO DAGScheduler - looking for newly runnable stages 15:45:49.545 INFO DAGScheduler - running: HashSet() 15:45:49.545 INFO DAGScheduler - waiting: HashSet(ResultStage 1241) 15:45:49.545 INFO DAGScheduler - failed: HashSet() 15:45:49.545 INFO DAGScheduler - Submitting ResultStage 1241 (MapPartitionsRDD[5521] at saveAsTextFile at SamSink.java:65), which has no missing parents 15:45:49.551 INFO MemoryStore - Block broadcast_2000 stored as values in memory (estimated size 160.3 KiB, free 1915.7 MiB) 15:45:49.552 INFO MemoryStore - Block broadcast_2000_piece0 stored as bytes in memory (estimated size 72.1 KiB, free 1915.6 MiB) 15:45:49.552 INFO BlockManagerInfo - Added broadcast_2000_piece0 in memory on localhost:43633 (size: 72.1 KiB, free: 1919.0 MiB) 15:45:49.552 INFO SparkContext - Created broadcast 2000 from broadcast at DAGScheduler.scala:1580 15:45:49.552 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 1241 (MapPartitionsRDD[5521] at saveAsTextFile at SamSink.java:65) (first 15 tasks are for partitions Vector(0)) 15:45:49.552 INFO TaskSchedulerImpl - Adding task set 1241.0 with 1 tasks resource profile 0 15:45:49.553 INFO TaskSetManager - Starting task 0.0 in stage 1241.0 (TID 1077) (localhost, executor driver, partition 0, NODE_LOCAL, 9376 bytes) 15:45:49.553 INFO Executor - Running task 0.0 in stage 1241.0 (TID 1077) 15:45:49.556 INFO ShuffleBlockFetcherIterator - Getting 1 (490.0 B) non-empty blocks including 1 (490.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks 15:45:49.556 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms 15:45:49.558 INFO HadoopMapRedCommitProtocol - Using output committer class org.apache.hadoop.mapred.FileOutputCommitter 15:45:49.558 INFO FileOutputCommitter - File Output Committer Algorithm version is 1 15:45:49.558 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false 15:45:49.569 INFO FileOutputCommitter - Saved output of task 'attempt_202505271545491118159094588045228_5521_m_000000_0' to file:/tmp/MarkDups10016148467204997135/MarkDups.sam.parts/_temporary/0/task_202505271545491118159094588045228_5521_m_000000 15:45:49.569 INFO SparkHadoopMapRedUtil - attempt_202505271545491118159094588045228_5521_m_000000_0: Committed. Elapsed time: 0 ms. 15:45:49.569 INFO Executor - Finished task 0.0 in stage 1241.0 (TID 1077). 1858 bytes result sent to driver 15:45:49.569 INFO TaskSetManager - Finished task 0.0 in stage 1241.0 (TID 1077) in 16 ms on localhost (executor driver) (1/1) 15:45:49.570 INFO TaskSchedulerImpl - Removed TaskSet 1241.0, whose tasks have all completed, from pool 15:45:49.570 INFO DAGScheduler - ResultStage 1241 (runJob at SparkHadoopWriter.scala:83) finished in 0.025 s 15:45:49.570 INFO DAGScheduler - Job 564 is finished. Cancelling potential speculative or zombie tasks for this job 15:45:49.570 INFO TaskSchedulerImpl - Killing all running tasks in stage 1241: Stage finished 15:45:49.570 INFO DAGScheduler - Job 564 finished: runJob at SparkHadoopWriter.scala:83, took 0.031645 s 15:45:49.570 INFO SparkHadoopWriter - Start to commit write Job job_202505271545491118159094588045228_5521. 15:45:49.574 INFO SparkHadoopWriter - Write Job job_202505271545491118159094588045228_5521 committed. Elapsed time: 4 ms. 15:45:49.581 INFO HadoopFileSystemWrapper - Concatenating 2 parts to /tmp/MarkDups10016148467204997135/MarkDups.sam 15:45:49.585 INFO HadoopFileSystemWrapper - Concatenating to /tmp/MarkDups10016148467204997135/MarkDups.sam done 15:45:49.585 INFO MarkDuplicatesSpark - Shutting down engine [May 27, 2025 at 3:45:49 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 15:45:49.590 INFO MarkDuplicatesSpark - ------------------------------------------------------------ 15:45:49.590 INFO MarkDuplicatesSpark - org.broadinstitute.hellbender.tools.spark.transforms.markduplicates vUnavailable 15:45:49.590 INFO MarkDuplicatesSpark - For support and documentation go to https://software.broadinstitute.org/gatk/ 15:45:49.590 INFO MarkDuplicatesSpark - Executing as runner@pkrvmf6wy0o8zjz on Linux v6.11.0-1014-azure amd64 15:45:49.590 INFO MarkDuplicatesSpark - Java runtime: OpenJDK 64-Bit Server VM v17.0.6+10 15:45:49.590 INFO MarkDuplicatesSpark - Start Date/Time: May 27, 2025 at 3:45:49 PM UTC 15:45:49.590 INFO MarkDuplicatesSpark - ------------------------------------------------------------ 15:45:49.590 INFO MarkDuplicatesSpark - ------------------------------------------------------------ 15:45:49.591 INFO MarkDuplicatesSpark - HTSJDK Defaults.COMPRESSION_LEVEL : 2 15:45:49.591 INFO MarkDuplicatesSpark - HTSJDK Defaults.USE_ASYNC_IO_READ_FOR_SAMTOOLS : false 15:45:49.591 INFO MarkDuplicatesSpark - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_SAMTOOLS : true 15:45:49.591 INFO MarkDuplicatesSpark - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_TRIBBLE : false 15:45:49.591 INFO MarkDuplicatesSpark - Deflater: IntelDeflater 15:45:49.591 INFO MarkDuplicatesSpark - Inflater: IntelInflater 15:45:49.591 INFO MarkDuplicatesSpark - GCS max retries/reopens: 20 15:45:49.591 INFO MarkDuplicatesSpark - Requester pays: disabled 15:45:49.591 INFO MarkDuplicatesSpark - Initializing engine 15:45:49.591 INFO MarkDuplicatesSpark - Done initializing engine 15:45:49.591 INFO MarkDuplicatesSpark - Spark verbosity set to INFO (see --spark-verbosity argument) 15:45:49.593 INFO MemoryStore - Block broadcast_2001 stored as values in memory (estimated size 306.3 KiB, free 1915.3 MiB) 15:45:49.599 INFO MemoryStore - Block broadcast_2001_piece0 stored as bytes in memory (estimated size 64.4 KiB, free 1915.3 MiB) 15:45:49.599 INFO BlockManagerInfo - Added broadcast_2001_piece0 in memory on localhost:43633 (size: 64.4 KiB, free: 1918.9 MiB) 15:45:49.600 INFO SparkContext - Created broadcast 2001 from newAPIHadoopFile at PathSplitSource.java:96 15:45:49.622 INFO MemoryStore - Block broadcast_2002 stored as values in memory (estimated size 306.3 KiB, free 1915.0 MiB) 15:45:49.628 INFO MemoryStore - Block broadcast_2002_piece0 stored as bytes in memory (estimated size 64.4 KiB, free 1914.9 MiB) 15:45:49.628 INFO BlockManagerInfo - Added broadcast_2002_piece0 in memory on localhost:43633 (size: 64.4 KiB, free: 1918.8 MiB) 15:45:49.628 INFO SparkContext - Created broadcast 2002 from newAPIHadoopFile at PathSplitSource.java:96 15:45:49.660 INFO SparkContext - Starting job: collect at SparkUtils.java:205 15:45:49.661 INFO FileInputFormat - Total input files to process : 1 15:45:49.661 INFO DAGScheduler - Registering RDD 5535 (mapToPair at SparkUtils.java:161) as input to shuffle 310 15:45:49.661 INFO DAGScheduler - Got job 565 (collect at SparkUtils.java:205) with 1 output partitions 15:45:49.661 INFO DAGScheduler - Final stage: ResultStage 1243 (collect at SparkUtils.java:205) 15:45:49.661 INFO DAGScheduler - Parents of final stage: List(ShuffleMapStage 1242) 15:45:49.661 INFO DAGScheduler - Missing parents: List(ShuffleMapStage 1242) 15:45:49.661 INFO DAGScheduler - Submitting ShuffleMapStage 1242 (MapPartitionsRDD[5535] at mapToPair at SparkUtils.java:161), which has no missing parents 15:45:49.678 INFO MemoryStore - Block broadcast_2003 stored as values in memory (estimated size 481.4 KiB, free 1914.4 MiB) 15:45:49.680 INFO MemoryStore - Block broadcast_2003_piece0 stored as bytes in memory (estimated size 207.4 KiB, free 1914.2 MiB) 15:45:49.680 INFO BlockManagerInfo - Added broadcast_2003_piece0 in memory on localhost:43633 (size: 207.4 KiB, free: 1918.6 MiB) 15:45:49.681 INFO SparkContext - Created broadcast 2003 from broadcast at DAGScheduler.scala:1580 15:45:49.681 INFO DAGScheduler - Submitting 1 missing tasks from ShuffleMapStage 1242 (MapPartitionsRDD[5535] at mapToPair at SparkUtils.java:161) (first 15 tasks are for partitions Vector(0)) 15:45:49.681 INFO TaskSchedulerImpl - Adding task set 1242.0 with 1 tasks resource profile 0 15:45:49.681 INFO TaskSetManager - Starting task 0.0 in stage 1242.0 (TID 1078) (localhost, executor driver, partition 0, PROCESS_LOCAL, 9757 bytes) 15:45:49.681 INFO Executor - Running task 0.0 in stage 1242.0 (TID 1078) 15:45:49.711 INFO NewHadoopRDD - Input split: file:/home/runner/work/gatk/gatk/src/test/resources/org/broadinstitute/hellbender/tools/walkers/MarkDuplicatesGATK/inputSingleLibrarySolexa16404.bam:0+5416 15:45:49.716 INFO Executor - Finished task 0.0 in stage 1242.0 (TID 1078). 1148 bytes result sent to driver 15:45:49.717 INFO TaskSetManager - Finished task 0.0 in stage 1242.0 (TID 1078) in 36 ms on localhost (executor driver) (1/1) 15:45:49.717 INFO TaskSchedulerImpl - Removed TaskSet 1242.0, whose tasks have all completed, from pool 15:45:49.717 INFO DAGScheduler - ShuffleMapStage 1242 (mapToPair at SparkUtils.java:161) finished in 0.056 s 15:45:49.717 INFO DAGScheduler - looking for newly runnable stages 15:45:49.717 INFO DAGScheduler - running: HashSet() 15:45:49.717 INFO DAGScheduler - waiting: HashSet(ResultStage 1243) 15:45:49.717 INFO DAGScheduler - failed: HashSet() 15:45:49.717 INFO DAGScheduler - Submitting ResultStage 1243 (MapPartitionsRDD[5538] at mapPartitions at SparkUtils.java:188), which has no missing parents 15:45:49.718 INFO MemoryStore - Block broadcast_2004 stored as values in memory (estimated size 7.4 KiB, free 1914.2 MiB) 15:45:49.718 INFO MemoryStore - Block broadcast_2004_piece0 stored as bytes in memory (estimated size 4.1 KiB, free 1914.2 MiB) 15:45:49.718 INFO BlockManagerInfo - Added broadcast_2004_piece0 in memory on localhost:43633 (size: 4.1 KiB, free: 1918.6 MiB) 15:45:49.718 INFO SparkContext - Created broadcast 2004 from broadcast at DAGScheduler.scala:1580 15:45:49.718 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 1243 (MapPartitionsRDD[5538] at mapPartitions at SparkUtils.java:188) (first 15 tasks are for partitions Vector(0)) 15:45:49.718 INFO TaskSchedulerImpl - Adding task set 1243.0 with 1 tasks resource profile 0 15:45:49.719 INFO TaskSetManager - Starting task 0.0 in stage 1243.0 (TID 1079) (localhost, executor driver, partition 0, NODE_LOCAL, 9376 bytes) 15:45:49.719 INFO Executor - Running task 0.0 in stage 1243.0 (TID 1079) 15:45:49.720 INFO ShuffleBlockFetcherIterator - Getting 1 (4.3 KiB) non-empty blocks including 1 (4.3 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks 15:45:49.720 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms 15:45:49.722 INFO Executor - Finished task 0.0 in stage 1243.0 (TID 1079). 2039 bytes result sent to driver 15:45:49.722 INFO TaskSetManager - Finished task 0.0 in stage 1243.0 (TID 1079) in 3 ms on localhost (executor driver) (1/1) 15:45:49.722 INFO TaskSchedulerImpl - Removed TaskSet 1243.0, whose tasks have all completed, from pool 15:45:49.722 INFO DAGScheduler - ResultStage 1243 (collect at SparkUtils.java:205) finished in 0.005 s 15:45:49.722 INFO DAGScheduler - Job 565 is finished. Cancelling potential speculative or zombie tasks for this job 15:45:49.722 INFO TaskSchedulerImpl - Killing all running tasks in stage 1243: Stage finished 15:45:49.723 INFO DAGScheduler - Job 565 finished: collect at SparkUtils.java:205, took 0.062128 s 15:45:49.727 INFO MemoryStore - Block broadcast_2005 stored as values in memory (estimated size 392.0 B, free 1914.2 MiB) 15:45:49.727 INFO MemoryStore - Block broadcast_2005_piece0 stored as bytes in memory (estimated size 33.0 B, free 1914.2 MiB) 15:45:49.727 INFO BlockManagerInfo - Added broadcast_2005_piece0 in memory on localhost:43633 (size: 33.0 B, free: 1918.6 MiB) 15:45:49.727 INFO SparkContext - Created broadcast 2005 from broadcast at MarkDuplicatesSparkUtils.java:126 15:45:49.727 INFO MemoryStore - Block broadcast_2006 stored as values in memory (estimated size 144.0 B, free 1914.2 MiB) 15:45:49.728 INFO MemoryStore - Block broadcast_2006_piece0 stored as bytes in memory (estimated size 28.0 B, free 1914.2 MiB) 15:45:49.728 INFO BlockManagerInfo - Added broadcast_2006_piece0 in memory on localhost:43633 (size: 28.0 B, free: 1918.6 MiB) 15:45:49.728 INFO SparkContext - Created broadcast 2006 from broadcast at MarkDuplicatesSparkUtils.java:127 15:45:49.741 INFO SparkContext - Starting job: collectAsMap at MarkDuplicatesSparkUtils.java:542 15:45:49.741 INFO DAGScheduler - Registering RDD 5547 (flatMapToPair at MarkDuplicatesSparkUtils.java:130) as input to shuffle 313 15:45:49.741 INFO DAGScheduler - Registering RDD 5551 (mapToPair at MarkDuplicatesSpark.java:217) as input to shuffle 312 15:45:49.741 INFO DAGScheduler - Registering RDD 5555 (mapToPair at MarkDuplicatesSparkUtils.java:497) as input to shuffle 311 15:45:49.741 INFO DAGScheduler - Got job 566 (collectAsMap at MarkDuplicatesSparkUtils.java:542) with 1 output partitions 15:45:49.741 INFO DAGScheduler - Final stage: ResultStage 1248 (collectAsMap at MarkDuplicatesSparkUtils.java:542) 15:45:49.741 INFO DAGScheduler - Parents of final stage: List(ShuffleMapStage 1247) 15:45:49.741 INFO DAGScheduler - Missing parents: List(ShuffleMapStage 1247) 15:45:49.741 INFO DAGScheduler - Submitting ShuffleMapStage 1245 (MapPartitionsRDD[5547] at flatMapToPair at MarkDuplicatesSparkUtils.java:130), which has no missing parents 15:45:49.742 INFO MemoryStore - Block broadcast_2007 stored as values in memory (estimated size 45.2 KiB, free 1914.2 MiB) 15:45:49.743 INFO MemoryStore - Block broadcast_2007_piece0 stored as bytes in memory (estimated size 17.1 KiB, free 1914.2 MiB) 15:45:49.743 INFO BlockManagerInfo - Added broadcast_2007_piece0 in memory on localhost:43633 (size: 17.1 KiB, free: 1918.6 MiB) 15:45:49.743 INFO SparkContext - Created broadcast 2007 from broadcast at DAGScheduler.scala:1580 15:45:49.743 INFO DAGScheduler - Submitting 1 missing tasks from ShuffleMapStage 1245 (MapPartitionsRDD[5547] at flatMapToPair at MarkDuplicatesSparkUtils.java:130) (first 15 tasks are for partitions Vector(0)) 15:45:49.743 INFO TaskSchedulerImpl - Adding task set 1245.0 with 1 tasks resource profile 0 15:45:49.743 INFO TaskSetManager - Starting task 0.0 in stage 1245.0 (TID 1080) (localhost, executor driver, partition 0, NODE_LOCAL, 9890 bytes) 15:45:49.743 INFO Executor - Running task 0.0 in stage 1245.0 (TID 1080) 15:45:49.745 INFO ShuffleBlockFetcherIterator - Getting 1 (4.3 KiB) non-empty blocks including 1 (4.3 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks 15:45:49.745 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms 15:45:49.747 INFO Executor - Finished task 0.0 in stage 1245.0 (TID 1080). 1922 bytes result sent to driver 15:45:49.747 INFO TaskSetManager - Finished task 0.0 in stage 1245.0 (TID 1080) in 4 ms on localhost (executor driver) (1/1) 15:45:49.747 INFO TaskSchedulerImpl - Removed TaskSet 1245.0, whose tasks have all completed, from pool 15:45:49.747 INFO DAGScheduler - ShuffleMapStage 1245 (flatMapToPair at MarkDuplicatesSparkUtils.java:130) finished in 0.005 s 15:45:49.747 INFO DAGScheduler - looking for newly runnable stages 15:45:49.747 INFO DAGScheduler - running: HashSet() 15:45:49.747 INFO DAGScheduler - waiting: HashSet(ResultStage 1248, ShuffleMapStage 1246, ShuffleMapStage 1247) 15:45:49.747 INFO DAGScheduler - failed: HashSet() 15:45:49.747 INFO DAGScheduler - Submitting ShuffleMapStage 1246 (MapPartitionsRDD[5551] at mapToPair at MarkDuplicatesSpark.java:217), which has no missing parents 15:45:49.748 INFO MemoryStore - Block broadcast_2008 stored as values in memory (estimated size 49.3 KiB, free 1914.1 MiB) 15:45:49.749 INFO MemoryStore - Block broadcast_2008_piece0 stored as bytes in memory (estimated size 19.3 KiB, free 1914.1 MiB) 15:45:49.749 INFO BlockManagerInfo - Added broadcast_2008_piece0 in memory on localhost:43633 (size: 19.3 KiB, free: 1918.6 MiB) 15:45:49.749 INFO SparkContext - Created broadcast 2008 from broadcast at DAGScheduler.scala:1580 15:45:49.749 INFO DAGScheduler - Submitting 1 missing tasks from ShuffleMapStage 1246 (MapPartitionsRDD[5551] at mapToPair at MarkDuplicatesSpark.java:217) (first 15 tasks are for partitions Vector(0)) 15:45:49.749 INFO TaskSchedulerImpl - Adding task set 1246.0 with 1 tasks resource profile 0 15:45:49.749 INFO TaskSetManager - Starting task 0.0 in stage 1246.0 (TID 1081) (localhost, executor driver, partition 0, NODE_LOCAL, 9365 bytes) 15:45:49.750 INFO Executor - Running task 0.0 in stage 1246.0 (TID 1081) 15:45:49.751 INFO ShuffleBlockFetcherIterator - Getting 1 (593.0 B) non-empty blocks including 1 (593.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks 15:45:49.751 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms 15:45:49.755 INFO Executor - Finished task 0.0 in stage 1246.0 (TID 1081). 1922 bytes result sent to driver 15:45:49.755 INFO TaskSetManager - Finished task 0.0 in stage 1246.0 (TID 1081) in 6 ms on localhost (executor driver) (1/1) 15:45:49.755 INFO TaskSchedulerImpl - Removed TaskSet 1246.0, whose tasks have all completed, from pool 15:45:49.755 INFO DAGScheduler - ShuffleMapStage 1246 (mapToPair at MarkDuplicatesSpark.java:217) finished in 0.007 s 15:45:49.755 INFO DAGScheduler - looking for newly runnable stages 15:45:49.755 INFO DAGScheduler - running: HashSet() 15:45:49.755 INFO DAGScheduler - waiting: HashSet(ResultStage 1248, ShuffleMapStage 1247) 15:45:49.755 INFO DAGScheduler - failed: HashSet() 15:45:49.755 INFO DAGScheduler - Submitting ShuffleMapStage 1247 (MapPartitionsRDD[5555] at mapToPair at MarkDuplicatesSparkUtils.java:497), which has no missing parents 15:45:49.756 INFO MemoryStore - Block broadcast_2009 stored as values in memory (estimated size 40.0 KiB, free 1914.0 MiB) 15:45:49.756 INFO MemoryStore - Block broadcast_2009_piece0 stored as bytes in memory (estimated size 14.5 KiB, free 1914.0 MiB) 15:45:49.756 INFO BlockManagerInfo - Added broadcast_2009_piece0 in memory on localhost:43633 (size: 14.5 KiB, free: 1918.6 MiB) 15:45:49.756 INFO SparkContext - Created broadcast 2009 from broadcast at DAGScheduler.scala:1580 15:45:49.757 INFO DAGScheduler - Submitting 1 missing tasks from ShuffleMapStage 1247 (MapPartitionsRDD[5555] at mapToPair at MarkDuplicatesSparkUtils.java:497) (first 15 tasks are for partitions Vector(0)) 15:45:49.757 INFO TaskSchedulerImpl - Adding task set 1247.0 with 1 tasks resource profile 0 15:45:49.757 INFO TaskSetManager - Starting task 0.0 in stage 1247.0 (TID 1082) (localhost, executor driver, partition 0, NODE_LOCAL, 9948 bytes) 15:45:49.757 INFO Executor - Running task 0.0 in stage 1247.0 (TID 1082) 15:45:49.758 INFO ShuffleBlockFetcherIterator - Getting 1 (4.3 KiB) non-empty blocks including 1 (4.3 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks 15:45:49.758 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms 15:45:49.759 INFO ShuffleBlockFetcherIterator - Getting 1 (652.0 B) non-empty blocks including 1 (652.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks 15:45:49.759 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms 15:45:49.760 INFO Executor - Finished task 0.0 in stage 1247.0 (TID 1082). 1879 bytes result sent to driver 15:45:49.761 INFO TaskSetManager - Finished task 0.0 in stage 1247.0 (TID 1082) in 4 ms on localhost (executor driver) (1/1) 15:45:49.761 INFO TaskSchedulerImpl - Removed TaskSet 1247.0, whose tasks have all completed, from pool 15:45:49.761 INFO DAGScheduler - ShuffleMapStage 1247 (mapToPair at MarkDuplicatesSparkUtils.java:497) finished in 0.006 s 15:45:49.761 INFO DAGScheduler - looking for newly runnable stages 15:45:49.761 INFO DAGScheduler - running: HashSet() 15:45:49.761 INFO DAGScheduler - waiting: HashSet(ResultStage 1248) 15:45:49.761 INFO DAGScheduler - failed: HashSet() 15:45:49.761 INFO DAGScheduler - Submitting ResultStage 1248 (MapPartitionsRDD[5557] at mapValues at MarkDuplicatesSparkUtils.java:519), which has no missing parents 15:45:49.761 INFO MemoryStore - Block broadcast_2010 stored as values in memory (estimated size 41.2 KiB, free 1914.0 MiB) 15:45:49.762 INFO MemoryStore - Block broadcast_2010_piece0 stored as bytes in memory (estimated size 15.3 KiB, free 1914.0 MiB) 15:45:49.762 INFO BlockManagerInfo - Added broadcast_2010_piece0 in memory on localhost:43633 (size: 15.3 KiB, free: 1918.6 MiB) 15:45:49.762 INFO SparkContext - Created broadcast 2010 from broadcast at DAGScheduler.scala:1580 15:45:49.762 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 1248 (MapPartitionsRDD[5557] at mapValues at MarkDuplicatesSparkUtils.java:519) (first 15 tasks are for partitions Vector(0)) 15:45:49.762 INFO TaskSchedulerImpl - Adding task set 1248.0 with 1 tasks resource profile 0 15:45:49.762 INFO TaskSetManager - Starting task 0.0 in stage 1248.0 (TID 1083) (localhost, executor driver, partition 0, NODE_LOCAL, 9376 bytes) 15:45:49.763 INFO Executor - Running task 0.0 in stage 1248.0 (TID 1083) 15:45:49.764 INFO ShuffleBlockFetcherIterator - Getting 1 (142.0 B) non-empty blocks including 1 (142.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks 15:45:49.764 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms 15:45:49.765 INFO Executor - Finished task 0.0 in stage 1248.0 (TID 1083). 1901 bytes result sent to driver 15:45:49.765 INFO TaskSetManager - Finished task 0.0 in stage 1248.0 (TID 1083) in 3 ms on localhost (executor driver) (1/1) 15:45:49.765 INFO TaskSchedulerImpl - Removed TaskSet 1248.0, whose tasks have all completed, from pool 15:45:49.765 INFO DAGScheduler - ResultStage 1248 (collectAsMap at MarkDuplicatesSparkUtils.java:542) finished in 0.004 s 15:45:49.765 INFO DAGScheduler - Job 566 is finished. Cancelling potential speculative or zombie tasks for this job 15:45:49.765 INFO TaskSchedulerImpl - Killing all running tasks in stage 1248: Stage finished 15:45:49.765 INFO DAGScheduler - Job 566 finished: collectAsMap at MarkDuplicatesSparkUtils.java:542, took 0.024617 s 15:45:49.776 INFO MemoryStore - Block broadcast_2011 stored as values in memory (estimated size 93.5 KiB, free 1913.9 MiB) 15:45:49.776 INFO MemoryStore - Block broadcast_2011_piece0 stored as bytes in memory (estimated size 5.6 KiB, free 1913.9 MiB) 15:45:49.777 INFO BlockManagerInfo - Added broadcast_2011_piece0 in memory on localhost:43633 (size: 5.6 KiB, free: 1918.6 MiB) 15:45:49.777 INFO SparkContext - Created broadcast 2011 from broadcast at ReadsSparkSink.java:133 15:45:49.780 INFO HadoopMapRedCommitProtocol - Using output committer class org.apache.hadoop.mapred.FileOutputCommitter 15:45:49.780 INFO FileOutputCommitter - File Output Committer Algorithm version is 1 15:45:49.780 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false 15:45:49.797 INFO SparkContext - Starting job: runJob at SparkHadoopWriter.scala:83 15:45:49.797 INFO DAGScheduler - Registering RDD 5559 (mapToPair at SparkUtils.java:161) as input to shuffle 314 15:45:49.798 INFO DAGScheduler - Got job 567 (runJob at SparkHadoopWriter.scala:83) with 1 output partitions 15:45:49.798 INFO DAGScheduler - Final stage: ResultStage 1253 (runJob at SparkHadoopWriter.scala:83) 15:45:49.798 INFO DAGScheduler - Parents of final stage: List(ShuffleMapStage 1252) 15:45:49.798 INFO DAGScheduler - Missing parents: List(ShuffleMapStage 1252) 15:45:49.798 INFO DAGScheduler - Submitting ShuffleMapStage 1252 (MapPartitionsRDD[5559] at mapToPair at SparkUtils.java:161), which has no missing parents 15:45:49.798 INFO MemoryStore - Block broadcast_2012 stored as values in memory (estimated size 39.1 KiB, free 1913.8 MiB) 15:45:49.799 INFO MemoryStore - Block broadcast_2012_piece0 stored as bytes in memory (estimated size 14.1 KiB, free 1913.8 MiB) 15:45:49.799 INFO BlockManagerInfo - Added broadcast_2012_piece0 in memory on localhost:43633 (size: 14.1 KiB, free: 1918.6 MiB) 15:45:49.799 INFO SparkContext - Created broadcast 2012 from broadcast at DAGScheduler.scala:1580 15:45:49.799 INFO DAGScheduler - Submitting 1 missing tasks from ShuffleMapStage 1252 (MapPartitionsRDD[5559] at mapToPair at SparkUtils.java:161) (first 15 tasks are for partitions Vector(0)) 15:45:49.799 INFO TaskSchedulerImpl - Adding task set 1252.0 with 1 tasks resource profile 0 15:45:49.800 INFO TaskSetManager - Starting task 0.0 in stage 1252.0 (TID 1084) (localhost, executor driver, partition 0, NODE_LOCAL, 9948 bytes) 15:45:49.800 INFO Executor - Running task 0.0 in stage 1252.0 (TID 1084) 15:45:49.801 INFO ShuffleBlockFetcherIterator - Getting 1 (4.3 KiB) non-empty blocks including 1 (4.3 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks 15:45:49.801 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms 15:45:49.803 INFO ShuffleBlockFetcherIterator - Getting 1 (652.0 B) non-empty blocks including 1 (652.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks 15:45:49.803 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms 15:45:49.804 INFO Executor - Finished task 0.0 in stage 1252.0 (TID 1084). 1922 bytes result sent to driver 15:45:49.804 INFO TaskSetManager - Finished task 0.0 in stage 1252.0 (TID 1084) in 4 ms on localhost (executor driver) (1/1) 15:45:49.804 INFO TaskSchedulerImpl - Removed TaskSet 1252.0, whose tasks have all completed, from pool 15:45:49.804 INFO DAGScheduler - ShuffleMapStage 1252 (mapToPair at SparkUtils.java:161) finished in 0.006 s 15:45:49.804 INFO DAGScheduler - looking for newly runnable stages 15:45:49.804 INFO DAGScheduler - running: HashSet() 15:45:49.804 INFO DAGScheduler - waiting: HashSet(ResultStage 1253) 15:45:49.804 INFO DAGScheduler - failed: HashSet() 15:45:49.804 INFO DAGScheduler - Submitting ResultStage 1253 (MapPartitionsRDD[5565] at saveAsTextFile at SamSink.java:65), which has no missing parents 15:45:49.810 INFO MemoryStore - Block broadcast_2013 stored as values in memory (estimated size 184.3 KiB, free 1913.6 MiB) 15:45:49.811 INFO MemoryStore - Block broadcast_2013_piece0 stored as bytes in memory (estimated size 78.3 KiB, free 1913.6 MiB) 15:45:49.812 INFO BlockManagerInfo - Added broadcast_2013_piece0 in memory on localhost:43633 (size: 78.3 KiB, free: 1918.5 MiB) 15:45:49.812 INFO SparkContext - Created broadcast 2013 from broadcast at DAGScheduler.scala:1580 15:45:49.812 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 1253 (MapPartitionsRDD[5565] at saveAsTextFile at SamSink.java:65) (first 15 tasks are for partitions Vector(0)) 15:45:49.812 INFO TaskSchedulerImpl - Adding task set 1253.0 with 1 tasks resource profile 0 15:45:49.812 INFO TaskSetManager - Starting task 0.0 in stage 1253.0 (TID 1085) (localhost, executor driver, partition 0, NODE_LOCAL, 9376 bytes) 15:45:49.812 INFO Executor - Running task 0.0 in stage 1253.0 (TID 1085) 15:45:49.815 INFO ShuffleBlockFetcherIterator - Getting 1 (3.9 KiB) non-empty blocks including 1 (3.9 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks 15:45:49.816 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms 15:45:49.817 INFO HadoopMapRedCommitProtocol - Using output committer class org.apache.hadoop.mapred.FileOutputCommitter 15:45:49.817 INFO FileOutputCommitter - File Output Committer Algorithm version is 1 15:45:49.817 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false 15:45:49.827 INFO FileOutputCommitter - Saved output of task 'attempt_202505271545492842531283843232672_5565_m_000000_0' to file:/tmp/MarkDups6159901249511369674/MarkDups.sam.parts/_temporary/0/task_202505271545492842531283843232672_5565_m_000000 15:45:49.828 INFO SparkHadoopMapRedUtil - attempt_202505271545492842531283843232672_5565_m_000000_0: Committed. Elapsed time: 0 ms. 15:45:49.828 INFO Executor - Finished task 0.0 in stage 1253.0 (TID 1085). 1858 bytes result sent to driver 15:45:49.828 INFO TaskSetManager - Finished task 0.0 in stage 1253.0 (TID 1085) in 16 ms on localhost (executor driver) (1/1) 15:45:49.828 INFO TaskSchedulerImpl - Removed TaskSet 1253.0, whose tasks have all completed, from pool 15:45:49.828 INFO DAGScheduler - ResultStage 1253 (runJob at SparkHadoopWriter.scala:83) finished in 0.024 s 15:45:49.828 INFO DAGScheduler - Job 567 is finished. Cancelling potential speculative or zombie tasks for this job 15:45:49.828 INFO TaskSchedulerImpl - Killing all running tasks in stage 1253: Stage finished 15:45:49.828 INFO DAGScheduler - Job 567 finished: runJob at SparkHadoopWriter.scala:83, took 0.031073 s 15:45:49.828 INFO SparkHadoopWriter - Start to commit write Job job_202505271545492842531283843232672_5565. 15:45:49.832 INFO SparkHadoopWriter - Write Job job_202505271545492842531283843232672_5565 committed. Elapsed time: 3 ms. 15:45:49.839 INFO HadoopFileSystemWrapper - Concatenating 2 parts to /tmp/MarkDups6159901249511369674/MarkDups.sam 15:45:49.843 INFO HadoopFileSystemWrapper - Concatenating to /tmp/MarkDups6159901249511369674/MarkDups.sam done 15:45:49.843 INFO MarkDuplicatesSpark - Shutting down engine [May 27, 2025 at 3:45:49 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:45:50 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:45:50 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:45:50 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:45:51 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:45:51 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:45:51 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:45:51 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:45:52 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:45:52 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:45:52 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:45:53 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:45:53 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:45:53 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:45:54 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:45:54 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:45:54 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:45:54 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:45:55 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 15:45:55.426 ERROR Executor - Exception in task 0.0 in stage 1473.0 (TID 1232) org.broadinstitute.hellbender.exceptions.UserException$ReadMissingReadGroup: SAM/BAM/CRAM file (unknown) is malformed: Read RUNID:7:1203:2886:82292 is missing the read group (RG) tag, which is required by the GATK. Please use http://gatkforums.broadinstitute.org/discussion/59/companion-utilities-replacereadgroups to fix this problem at org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSparkUtils.getLibraryForRead(MarkDuplicatesSparkUtils.java:64) ~[main/:?] at org.broadinstitute.hellbender.utils.read.markduplicates.sparkrecords.EmptyFragment.<init>(EmptyFragment.java:36) ~[main/:?] at org.broadinstitute.hellbender.utils.read.markduplicates.sparkrecords.MarkDuplicatesSparkRecord.newEmptyFragment(MarkDuplicatesSparkRecord.java:44) ~[main/:?] at org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSparkUtils.lambda$transformToDuplicateNames$0(MarkDuplicatesSparkUtils.java:141) ~[main/:?] at java.base/java.util.stream.ReferencePipeline$15$1.accept(ReferencePipeline.java:540) ~[?:?] at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625) ~[?:?] at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) ~[?:?] at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[?:?] at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921) ~[?:?] at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?] at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682) ~[?:?] at org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSparkUtils.lambda$transformToDuplicateNames$5ac2632f$1(MarkDuplicatesSparkUtils.java:150) ~[main/:?] at org.apache.spark.api.java.JavaRDDLike.$anonfun$flatMapToPair$1(JavaRDDLike.scala:143) ~[spark-core_2.13-3.5.0.jar:3.5.0] at scala.collection.Iterator$$anon$10.nextCur(Iterator.scala:594) ~[scala-library-2.13.14.jar:?] at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:608) ~[scala-library-2.13.14.jar:?] at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:140) ~[spark-core_2.13-3.5.0.jar:3.5.0] at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59) ~[spark-core_2.13-3.5.0.jar:3.5.0] at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:104) ~[spark-core_2.13-3.5.0.jar:3.5.0] at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:54) ~[spark-core_2.13-3.5.0.jar:3.5.0] at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:161) ~[spark-core_2.13-3.5.0.jar:3.5.0] at org.apache.spark.scheduler.Task.run(Task.scala:141) ~[spark-core_2.13-3.5.0.jar:3.5.0] at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:620) ~[spark-core_2.13-3.5.0.jar:3.5.0] at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64) ~[spark-common-utils_2.13-3.5.0.jar:3.5.0] at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61) ~[spark-common-utils_2.13-3.5.0.jar:3.5.0] at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94) ~[spark-core_2.13-3.5.0.jar:3.5.0] at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:623) [spark-core_2.13-3.5.0.jar:3.5.0] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?] at java.base/java.lang.Thread.run(Thread.java:833) [?:?] 15:45:55.443 ERROR TaskSetManager - Task 0 in stage 1473.0 failed 1 times; aborting job [May 27, 2025 at 3:45:55 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 15:45:55.655 ERROR Executor - Exception in task 0.0 in stage 1480.0 (TID 1235) org.broadinstitute.hellbender.exceptions.UserException$HeaderMissingReadGroup: SAM/BAM/CRAM file (unknown) is malformed: Read RUNID:7:1203:2886:82292 contains an (RG) tag with the group NotADuplicateGroup which is not found in the file header. at org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSparkUtils.getLibraryForRead(MarkDuplicatesSparkUtils.java:66) ~[main/:?] at org.broadinstitute.hellbender.utils.read.markduplicates.sparkrecords.EmptyFragment.<init>(EmptyFragment.java:36) ~[main/:?] at org.broadinstitute.hellbender.utils.read.markduplicates.sparkrecords.MarkDuplicatesSparkRecord.newEmptyFragment(MarkDuplicatesSparkRecord.java:44) ~[main/:?] at org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSparkUtils.lambda$transformToDuplicateNames$0(MarkDuplicatesSparkUtils.java:141) ~[main/:?] at java.base/java.util.stream.ReferencePipeline$15$1.accept(ReferencePipeline.java:540) ~[?:?] at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625) ~[?:?] at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) ~[?:?] at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[?:?] at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921) ~[?:?] at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?] at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682) ~[?:?] at org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSparkUtils.lambda$transformToDuplicateNames$5ac2632f$1(MarkDuplicatesSparkUtils.java:150) ~[main/:?] at org.apache.spark.api.java.JavaRDDLike.$anonfun$flatMapToPair$1(JavaRDDLike.scala:143) ~[spark-core_2.13-3.5.0.jar:3.5.0] at scala.collection.Iterator$$anon$10.nextCur(Iterator.scala:594) ~[scala-library-2.13.14.jar:?] at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:608) ~[scala-library-2.13.14.jar:?] at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:140) ~[spark-core_2.13-3.5.0.jar:3.5.0] at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59) ~[spark-core_2.13-3.5.0.jar:3.5.0] at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:104) ~[spark-core_2.13-3.5.0.jar:3.5.0] at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:54) ~[spark-core_2.13-3.5.0.jar:3.5.0] at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:161) ~[spark-core_2.13-3.5.0.jar:3.5.0] at org.apache.spark.scheduler.Task.run(Task.scala:141) ~[spark-core_2.13-3.5.0.jar:3.5.0] at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:620) ~[spark-core_2.13-3.5.0.jar:3.5.0] at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64) ~[spark-common-utils_2.13-3.5.0.jar:3.5.0] at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61) ~[spark-common-utils_2.13-3.5.0.jar:3.5.0] at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94) ~[spark-core_2.13-3.5.0.jar:3.5.0] at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:623) [spark-core_2.13-3.5.0.jar:3.5.0] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?] at java.base/java.lang.Thread.run(Thread.java:833) [?:?] 15:45:55.657 ERROR TaskSetManager - Task 0 in stage 1480.0 failed 1 times; aborting job [May 27, 2025 at 3:45:55 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:45:55 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:45:56 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:45:56 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:45:56 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:45:57 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:45:57 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:45:57 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:45:58 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:45:58 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:45:58 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:45:58 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:45:59 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:45:59 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:45:59 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:46:00 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:46:00 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:46:00 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:46:01 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:46:01 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:46:01 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:46:01 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:46:02 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:46:02 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:46:02 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:46:03 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:46:03 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:46:03 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:46:04 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:46:04 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:46:04 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:46:04 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:46:05 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:46:05 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:46:05 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:46:06 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:46:06 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:46:06 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:46:07 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:46:07 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:46:07 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:46:07 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:46:08 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:46:08 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:46:08 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:46:09 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:46:09 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:46:09 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:46:10 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:46:10 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:46:10 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:46:10 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:46:11 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:46:11 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:46:11 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:46:12 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:46:12 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:46:12 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:46:12 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:46:13 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:46:13 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:46:13 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:46:14 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:46:14 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:46:14 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:46:15 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:46:15 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:46:15 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:46:15 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:46:16 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:46:16 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:46:16 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:46:17 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:46:17 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:46:17 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:46:17 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:46:18 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:46:18 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:46:18 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:46:19 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:46:19 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360 [May 27, 2025 at 3:46:19 PM UTC] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 0.00 minutes. Runtime.totalMemory()=1426063360