Class org.broadinstitute.hellbender.engine.spark.datasources.VariantsSparkSinkUnitTest

32

tests

0

failures

0

ignored

22.854s

duration

100%

successful

Tests

Test Duration Result
testBrokenGVCFCasesAreDisallowed[0](false, .bcf) 0s passed
testBrokenGVCFCasesAreDisallowed[1](false, .bcf.gz) 0s passed
testBrokenGVCFCasesAreDisallowed[2](true, .g.bcf) 0s passed
testBrokenGVCFCasesAreDisallowed[3](true, .g.bcf.gz) 0s passed
testEnableDisableGVCFWriting[0](false, .vcf) 0.207s passed
testEnableDisableGVCFWriting[1](false, .vcf.gz) 0.211s passed
testEnableDisableGVCFWriting[2](true, .g.vcf) 0.200s passed
testEnableDisableGVCFWriting[3](true, .g.vcf.gz) 0.202s passed
testWritingToAnExistingFileHDFS[0](/home/runner/work/gatk/gatk/src/test/resources/Homo_sapiens_assembly19.dbsnp135.chr1_1M.exome_intervals.vcf, .vcf, true) 1.234s passed
testWritingToAnExistingFileHDFS[1](/home/runner/work/gatk/gatk/src/test/resources/Homo_sapiens_assembly19.dbsnp135.chr1_1M.exome_intervals.vcf, .vcf.bgz, true) 2.372s passed
testWritingToAnExistingFileHDFS[2](/home/runner/work/gatk/gatk/src/test/resources/Homo_sapiens_assembly19.dbsnp135.chr1_1M.exome_intervals.vcf, .vcf.bgz, false) 1.515s passed
testWritingToAnExistingFileHDFS[3](/home/runner/work/gatk/gatk/src/test/resources/Homo_sapiens_assembly19.dbsnp135.chr1_1M.exome_intervals.vcf, .vcf.gz, true) 2.346s passed
testWritingToAnExistingFileHDFS[4](/home/runner/work/gatk/gatk/src/test/resources/Homo_sapiens_assembly19.dbsnp135.chr1_1M.exome_intervals.vcf, .vcf.gz, false) 1.487s passed
testWritingToAnExistingFileHDFS[5](/home/runner/work/gatk/gatk/src/test/resources/HSA19.dbsnp135.chr1_1M.exome_intervals.modified.vcf, .vcf, true) 1.075s passed
testWritingToFileURL[0](/home/runner/work/gatk/gatk/src/test/resources/Homo_sapiens_assembly19.dbsnp135.chr1_1M.exome_intervals.vcf, .vcf, true) 0.291s passed
testWritingToFileURL[1](/home/runner/work/gatk/gatk/src/test/resources/Homo_sapiens_assembly19.dbsnp135.chr1_1M.exome_intervals.vcf, .vcf.bgz, true) 0.296s passed
testWritingToFileURL[2](/home/runner/work/gatk/gatk/src/test/resources/Homo_sapiens_assembly19.dbsnp135.chr1_1M.exome_intervals.vcf, .vcf.bgz, false) 0.288s passed
testWritingToFileURL[3](/home/runner/work/gatk/gatk/src/test/resources/Homo_sapiens_assembly19.dbsnp135.chr1_1M.exome_intervals.vcf, .vcf.gz, true) 0.292s passed
testWritingToFileURL[4](/home/runner/work/gatk/gatk/src/test/resources/Homo_sapiens_assembly19.dbsnp135.chr1_1M.exome_intervals.vcf, .vcf.gz, false) 0.269s passed
testWritingToFileURL[5](/home/runner/work/gatk/gatk/src/test/resources/HSA19.dbsnp135.chr1_1M.exome_intervals.modified.vcf, .vcf, true) 0.274s passed
variantsSinkHDFSTest[0](/home/runner/work/gatk/gatk/src/test/resources/Homo_sapiens_assembly19.dbsnp135.chr1_1M.exome_intervals.vcf, .vcf, true) 0.641s passed
variantsSinkHDFSTest[1](/home/runner/work/gatk/gatk/src/test/resources/Homo_sapiens_assembly19.dbsnp135.chr1_1M.exome_intervals.vcf, .vcf.bgz, true) 2.264s passed
variantsSinkHDFSTest[2](/home/runner/work/gatk/gatk/src/test/resources/Homo_sapiens_assembly19.dbsnp135.chr1_1M.exome_intervals.vcf, .vcf.bgz, false) 1.489s passed
variantsSinkHDFSTest[3](/home/runner/work/gatk/gatk/src/test/resources/Homo_sapiens_assembly19.dbsnp135.chr1_1M.exome_intervals.vcf, .vcf.gz, true) 1.880s passed
variantsSinkHDFSTest[4](/home/runner/work/gatk/gatk/src/test/resources/Homo_sapiens_assembly19.dbsnp135.chr1_1M.exome_intervals.vcf, .vcf.gz, false) 1.474s passed
variantsSinkHDFSTest[5](/home/runner/work/gatk/gatk/src/test/resources/HSA19.dbsnp135.chr1_1M.exome_intervals.modified.vcf, .vcf, true) 1.052s passed
variantsSinkTest[0](/home/runner/work/gatk/gatk/src/test/resources/Homo_sapiens_assembly19.dbsnp135.chr1_1M.exome_intervals.vcf, .vcf, true) 0.235s passed
variantsSinkTest[1](/home/runner/work/gatk/gatk/src/test/resources/Homo_sapiens_assembly19.dbsnp135.chr1_1M.exome_intervals.vcf, .vcf.bgz, true) 0.273s passed
variantsSinkTest[2](/home/runner/work/gatk/gatk/src/test/resources/Homo_sapiens_assembly19.dbsnp135.chr1_1M.exome_intervals.vcf, .vcf.bgz, false) 0.249s passed
variantsSinkTest[3](/home/runner/work/gatk/gatk/src/test/resources/Homo_sapiens_assembly19.dbsnp135.chr1_1M.exome_intervals.vcf, .vcf.gz, true) 0.247s passed
variantsSinkTest[4](/home/runner/work/gatk/gatk/src/test/resources/Homo_sapiens_assembly19.dbsnp135.chr1_1M.exome_intervals.vcf, .vcf.gz, false) 0.259s passed
variantsSinkTest[5](/home/runner/work/gatk/gatk/src/test/resources/HSA19.dbsnp135.chr1_1M.exome_intervals.modified.vcf, .vcf, true) 0.232s passed

Standard error

14:52:02.823 INFO  MiniDFSCluster - starting cluster: numNameNodes=1, numDataNodes=1
14:52:02.824 INFO  NameNode - Formatting using clusterid: testClusterID
14:52:02.824 INFO  FSEditLog - Edit logging is async:true
14:52:02.832 INFO  FSNamesystem - KeyProvider: null
14:52:02.832 INFO  FSNamesystem - fsLock is fair: true
14:52:02.832 INFO  FSNamesystem - Detailed lock hold time metrics enabled: false
14:52:02.832 INFO  FSNamesystem - fsOwner                = runner (auth:SIMPLE)
14:52:02.832 INFO  FSNamesystem - supergroup             = supergroup
14:52:02.832 INFO  FSNamesystem - isPermissionEnabled    = true
14:52:02.832 INFO  FSNamesystem - isStoragePolicyEnabled = true
14:52:02.832 INFO  FSNamesystem - HA Enabled: false
14:52:02.832 INFO  Util - dfs.datanode.fileio.profiling.sampling.percentage set to 0. Disabling file IO profiling
14:52:02.832 INFO  DatanodeManager - dfs.block.invalidate.limit : configured=1000, counted=60, effected=1000
14:52:02.832 INFO  DatanodeManager - dfs.namenode.datanode.registration.ip-hostname-check=true
14:52:02.832 INFO  BlockManager - dfs.namenode.startup.delay.block.deletion.sec is set to 000:00:00:00.000
14:52:02.832 INFO  BlockManager - The block deletion will start around 2026 Mar 04 14:52:02
14:52:02.833 INFO  GSet - Computing capacity for map BlocksMap
14:52:02.833 INFO  GSet - VM type       = 64-bit
14:52:02.833 INFO  GSet - 2.0% max memory 3.4 GB = 70 MB
14:52:02.833 INFO  GSet - capacity      = 2^23 = 8388608 entries
14:52:02.847 INFO  BlockManager - Storage policy satisfier is disabled
14:52:02.847 INFO  BlockManager - dfs.block.access.token.enable = false
14:52:02.847 INFO  BlockManagerSafeMode - dfs.namenode.safemode.threshold-pct = 0.999
14:52:02.847 INFO  BlockManagerSafeMode - dfs.namenode.safemode.min.datanodes = 0
14:52:02.847 INFO  BlockManagerSafeMode - dfs.namenode.safemode.extension = 0
14:52:02.847 INFO  BlockManager - defaultReplication         = 1
14:52:02.847 INFO  BlockManager - maxReplication             = 512
14:52:02.847 INFO  BlockManager - minReplication             = 1
14:52:02.847 INFO  BlockManager - maxReplicationStreams      = 2
14:52:02.847 INFO  BlockManager - redundancyRecheckInterval  = 3000ms
14:52:02.847 INFO  BlockManager - encryptDataTransfer        = false
14:52:02.847 INFO  BlockManager - maxNumBlocksToLog          = 1000
14:52:02.847 INFO  GSet - Computing capacity for map INodeMap
14:52:02.847 INFO  GSet - VM type       = 64-bit
14:52:02.847 INFO  GSet - 1.0% max memory 3.4 GB = 35 MB
14:52:02.847 INFO  GSet - capacity      = 2^22 = 4194304 entries
14:52:02.854 INFO  FSDirectory - ACLs enabled? true
14:52:02.854 INFO  FSDirectory - POSIX ACL inheritance enabled? true
14:52:02.854 INFO  FSDirectory - XAttrs enabled? true
14:52:02.854 INFO  NameNode - Caching file names occurring more than 10 times
14:52:02.854 INFO  SnapshotManager - Loaded config captureOpenFiles: false, skipCaptureAccessTimeOnlyChange: false, snapshotDiffAllowSnapRootDescendant: true, maxSnapshotLimit: 65536
14:52:02.854 INFO  SnapshotManager - SkipList is disabled
14:52:02.854 INFO  GSet - Computing capacity for map cachedBlocks
14:52:02.854 INFO  GSet - VM type       = 64-bit
14:52:02.855 INFO  GSet - 0.25% max memory 3.4 GB = 8.8 MB
14:52:02.855 INFO  GSet - capacity      = 2^20 = 1048576 entries
14:52:02.860 INFO  TopMetrics - NNTop conf: dfs.namenode.top.window.num.buckets = 10
14:52:02.860 INFO  TopMetrics - NNTop conf: dfs.namenode.top.num.users = 10
14:52:02.860 INFO  TopMetrics - NNTop conf: dfs.namenode.top.windows.minutes = 1,5,25
14:52:02.860 INFO  FSNamesystem - Retry cache on namenode is enabled
14:52:02.860 INFO  FSNamesystem - Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis
14:52:02.860 INFO  GSet - Computing capacity for map NameNodeRetryCache
14:52:02.860 INFO  GSet - VM type       = 64-bit
14:52:02.860 INFO  GSet - 0.029999999329447746% max memory 3.4 GB = 1.0 MB
14:52:02.860 INFO  GSet - capacity      = 2^17 = 131072 entries
14:52:02.860 INFO  FSImage - Allocated new BlockPoolId: BP-1183507661-10.1.0.125-1772635922860
14:52:02.862 INFO  Storage - Storage directory /tmp/minicluster_storage1617096220754514403/name-0-1 has been successfully formatted.
14:52:02.863 INFO  Storage - Storage directory /tmp/minicluster_storage1617096220754514403/name-0-2 has been successfully formatted.
14:52:02.876 INFO  FSImageFormatProtobuf - Saving image file /tmp/minicluster_storage1617096220754514403/name-0-1/current/fsimage.ckpt_0000000000000000000 using no compression
14:52:02.876 INFO  FSImageFormatProtobuf - Saving image file /tmp/minicluster_storage1617096220754514403/name-0-2/current/fsimage.ckpt_0000000000000000000 using no compression
14:52:02.880 INFO  FSImageFormatProtobuf - Image file /tmp/minicluster_storage1617096220754514403/name-0-1/current/fsimage.ckpt_0000000000000000000 of size 419 bytes saved in 0 seconds .
14:52:02.880 INFO  FSImageFormatProtobuf - Image file /tmp/minicluster_storage1617096220754514403/name-0-2/current/fsimage.ckpt_0000000000000000000 of size 419 bytes saved in 0 seconds .
14:52:02.881 INFO  NNStorageRetentionManager - Going to retain 1 images with txid >= 0
14:52:02.884 INFO  FSNamesystem - Stopping services started for active state
14:52:02.884 INFO  FSNamesystem - Stopping services started for standby state
14:52:02.885 INFO  NameNode - createNameNode []
14:52:02.886 WARN  MetricsConfig - Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
14:52:02.886 INFO  MetricsSystemImpl - Scheduled Metric snapshot period at 10 second(s).
14:52:02.886 INFO  MetricsSystemImpl - NameNode metrics system started
14:52:02.887 INFO  NameNodeUtils - fs.defaultFS is hdfs://127.0.0.1:0
14:52:02.897 INFO  JvmPauseMonitor - Starting JVM pause monitor
14:52:02.897 INFO  DFSUtil - Filter initializers set : org.apache.hadoop.http.lib.StaticUserWebFilter,org.apache.hadoop.hdfs.web.AuthFilterInitializer
14:52:02.897 INFO  DFSUtil - Starting Web-server for hdfs at: http://localhost:0
14:52:02.899 WARN  AuthenticationFilter - Unable to initialize FileSignerSecretProvider, falling back to use random secrets. Reason: Could not read signature secret file: /home/runner/hadoop-http-auth-signature-secret
14:52:02.899 WARN  HttpRequestLog - Jetty request log can only be enabled using Log4j
14:52:02.900 INFO  HttpServer2 - Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)
14:52:02.900 INFO  HttpServer2 - Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context hdfs
14:52:02.900 INFO  HttpServer2 - Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context static
14:52:02.902 INFO  HttpServer2 - Added filter AuthFilter (class=org.apache.hadoop.hdfs.web.AuthFilter) to context hdfs
14:52:02.902 INFO  HttpServer2 - Added filter AuthFilter (class=org.apache.hadoop.hdfs.web.AuthFilter) to context static
14:52:02.902 INFO  HttpServer2 - addJerseyResourcePackage: packageName=org.apache.hadoop.hdfs.server.namenode.web.resources;org.apache.hadoop.hdfs.web.resources, pathSpec=/webhdfs/v1/*
14:52:02.902 INFO  HttpServer2 - Jetty bound to port 41953
14:52:02.902 INFO  Server - jetty-9.4.56.v20240826; built: 2024-08-26T17:15:05.868Z; git: ec6782ff5ead824dabdcf47fa98f90a4aedff401; jvm 17.0.6+10
14:52:02.904 INFO  session - DefaultSessionIdManager workerName=node0
14:52:02.904 INFO  session - No SessionScavenger set, using defaults
14:52:02.904 INFO  session - node0 Scavenging every 660000ms
14:52:02.904 WARN  AuthenticationFilter - Unable to initialize FileSignerSecretProvider, falling back to use random secrets. Reason: Could not read signature secret file: /home/runner/hadoop-http-auth-signature-secret
14:52:02.904 INFO  ContextHandler - Started o.e.j.s.ServletContextHandler@5176ae1c{static,/static,jar:file:/home/runner/.gradle/caches/modules-2/files-2.1/org.apache.hadoop/hadoop-hdfs/3.3.6/5058b645375c6a68f509e167ad6a6ada9642df09/hadoop-hdfs-3.3.6-tests.jar!/webapps/static,AVAILABLE}
14:52:02.991 INFO  ContextHandler - Started o.e.j.w.WebAppContext@6a665bb8{hdfs,/,file:///tmp/jetty-localhost-41953-hadoop-hdfs-3_3_6-tests_jar-_-any-5159945223181492921/webapp/,AVAILABLE}{jar:file:/home/runner/.gradle/caches/modules-2/files-2.1/org.apache.hadoop/hadoop-hdfs/3.3.6/5058b645375c6a68f509e167ad6a6ada9642df09/hadoop-hdfs-3.3.6-tests.jar!/webapps/hdfs}
14:52:02.992 INFO  AbstractConnector - Started ServerConnector@789bc477{HTTP/1.1, (http/1.1)}{localhost:41953}
14:52:02.992 INFO  Server - Started @82359ms
14:52:02.993 INFO  FSEditLog - Edit logging is async:true
14:52:03.002 INFO  FSNamesystem - KeyProvider: null
14:52:03.002 INFO  FSNamesystem - fsLock is fair: true
14:52:03.002 INFO  FSNamesystem - Detailed lock hold time metrics enabled: false
14:52:03.002 INFO  FSNamesystem - fsOwner                = runner (auth:SIMPLE)
14:52:03.002 INFO  FSNamesystem - supergroup             = supergroup
14:52:03.002 INFO  FSNamesystem - isPermissionEnabled    = true
14:52:03.002 INFO  FSNamesystem - isStoragePolicyEnabled = true
14:52:03.002 INFO  FSNamesystem - HA Enabled: false
14:52:03.003 INFO  Util - dfs.datanode.fileio.profiling.sampling.percentage set to 0. Disabling file IO profiling
14:52:03.003 INFO  DatanodeManager - dfs.block.invalidate.limit : configured=1000, counted=60, effected=1000
14:52:03.003 INFO  DatanodeManager - dfs.namenode.datanode.registration.ip-hostname-check=true
14:52:03.003 INFO  BlockManager - dfs.namenode.startup.delay.block.deletion.sec is set to 000:00:00:00.000
14:52:03.003 INFO  BlockManager - The block deletion will start around 2026 Mar 04 14:52:03
14:52:03.003 INFO  GSet - Computing capacity for map BlocksMap
14:52:03.003 INFO  GSet - VM type       = 64-bit
14:52:03.003 INFO  GSet - 2.0% max memory 3.4 GB = 70 MB
14:52:03.003 INFO  GSet - capacity      = 2^23 = 8388608 entries
14:52:03.012 INFO  BlockManager - Storage policy satisfier is disabled
14:52:03.012 INFO  BlockManager - dfs.block.access.token.enable = false
14:52:03.012 INFO  BlockManagerSafeMode - dfs.namenode.safemode.threshold-pct = 0.999
14:52:03.012 INFO  BlockManagerSafeMode - dfs.namenode.safemode.min.datanodes = 0
14:52:03.012 INFO  BlockManagerSafeMode - dfs.namenode.safemode.extension = 0
14:52:03.012 INFO  BlockManager - defaultReplication         = 1
14:52:03.012 INFO  BlockManager - maxReplication             = 512
14:52:03.012 INFO  BlockManager - minReplication             = 1
14:52:03.012 INFO  BlockManager - maxReplicationStreams      = 2
14:52:03.012 INFO  BlockManager - redundancyRecheckInterval  = 3000ms
14:52:03.012 INFO  BlockManager - encryptDataTransfer        = false
14:52:03.012 INFO  BlockManager - maxNumBlocksToLog          = 1000
14:52:03.012 INFO  GSet - Computing capacity for map INodeMap
14:52:03.012 INFO  GSet - VM type       = 64-bit
14:52:03.012 INFO  GSet - 1.0% max memory 3.4 GB = 35 MB
14:52:03.012 INFO  GSet - capacity      = 2^22 = 4194304 entries
14:52:03.013 INFO  FSDirectory - ACLs enabled? true
14:52:03.013 INFO  FSDirectory - POSIX ACL inheritance enabled? true
14:52:03.014 INFO  FSDirectory - XAttrs enabled? true
14:52:03.014 INFO  NameNode - Caching file names occurring more than 10 times
14:52:03.014 INFO  SnapshotManager - Loaded config captureOpenFiles: false, skipCaptureAccessTimeOnlyChange: false, snapshotDiffAllowSnapRootDescendant: true, maxSnapshotLimit: 65536
14:52:03.014 INFO  SnapshotManager - SkipList is disabled
14:52:03.014 INFO  GSet - Computing capacity for map cachedBlocks
14:52:03.014 INFO  GSet - VM type       = 64-bit
14:52:03.014 INFO  GSet - 0.25% max memory 3.4 GB = 8.8 MB
14:52:03.014 INFO  GSet - capacity      = 2^20 = 1048576 entries
14:52:03.014 INFO  TopMetrics - NNTop conf: dfs.namenode.top.window.num.buckets = 10
14:52:03.014 INFO  TopMetrics - NNTop conf: dfs.namenode.top.num.users = 10
14:52:03.014 INFO  TopMetrics - NNTop conf: dfs.namenode.top.windows.minutes = 1,5,25
14:52:03.014 INFO  FSNamesystem - Retry cache on namenode is enabled
14:52:03.014 INFO  FSNamesystem - Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis
14:52:03.014 INFO  GSet - Computing capacity for map NameNodeRetryCache
14:52:03.014 INFO  GSet - VM type       = 64-bit
14:52:03.014 INFO  GSet - 0.029999999329447746% max memory 3.4 GB = 1.0 MB
14:52:03.014 INFO  GSet - capacity      = 2^17 = 131072 entries
14:52:03.016 INFO  Storage - Lock on /tmp/minicluster_storage1617096220754514403/name-0-1/in_use.lock acquired by nodename 3348@runnervmnay03
14:52:03.016 INFO  Storage - Lock on /tmp/minicluster_storage1617096220754514403/name-0-2/in_use.lock acquired by nodename 3348@runnervmnay03
14:52:03.017 INFO  FileJournalManager - Recovering unfinalized segments in /tmp/minicluster_storage1617096220754514403/name-0-1/current
14:52:03.017 INFO  FileJournalManager - Recovering unfinalized segments in /tmp/minicluster_storage1617096220754514403/name-0-2/current
14:52:03.017 INFO  FSImage - No edit log streams selected.
14:52:03.017 INFO  FSImage - Planning to load image: FSImageFile(file=/tmp/minicluster_storage1617096220754514403/name-0-1/current/fsimage_0000000000000000000, cpktTxId=0000000000000000000)
14:52:03.018 INFO  FSImageFormatPBINode - Loading 1 INodes.
14:52:03.018 INFO  FSImageFormatPBINode - Successfully loaded 1 inodes
14:52:03.018 INFO  FSImageFormatPBINode - Completed update blocks map and name cache, total waiting duration 0ms.
14:52:03.018 INFO  FSImageFormatProtobuf - Loaded FSImage in 0 seconds.
14:52:03.018 INFO  FSImage - Loaded image for txid 0 from /tmp/minicluster_storage1617096220754514403/name-0-1/current/fsimage_0000000000000000000
14:52:03.018 INFO  FSNamesystem - Need to save fs image? false (staleImage=false, haEnabled=false, isRollingUpgrade=false)
14:52:03.019 INFO  FSEditLog - Starting log segment at 1
14:52:03.023 INFO  NameCache - initialized with 0 entries 0 lookups
14:52:03.023 INFO  FSNamesystem - Finished loading FSImage in 8 msecs
14:52:03.023 INFO  NameNode - RPC server is binding to localhost:0
14:52:03.023 INFO  NameNode - Enable NameNode state context:false
14:52:03.024 INFO  CallQueueManager - Using callQueue: class java.util.concurrent.LinkedBlockingQueue, queueCapacity: 1000, scheduler: class org.apache.hadoop.ipc.DefaultRpcScheduler, ipcBackoff: false.
14:52:03.024 INFO  Server - Listener at localhost:42535
14:52:03.024 INFO  Server - Starting Socket Reader #1 for port 0
14:52:03.026 INFO  NameNode - Clients are to use localhost:42535 to access this namenode/service.
14:52:03.026 INFO  FSNamesystem - Registered FSNamesystemState, ReplicatedBlocksState and ECBlockGroupsState MBeans.
14:52:03.039 INFO  LeaseManager - Number of blocks under construction: 0
14:52:03.039 INFO  DatanodeAdminDefaultMonitor - Initialized the Default Decommission and Maintenance monitor
14:52:03.040 INFO  BlockManager - Start MarkedDeleteBlockScrubber thread
14:52:03.040 INFO  BlockManager - initializing replication queues
14:52:03.040 INFO  StateChange - STATE* Leaving safe mode after 0 secs
14:52:03.040 INFO  StateChange - STATE* Network topology has 0 racks and 0 datanodes
14:52:03.040 INFO  StateChange - STATE* UnderReplicatedBlocks has 0 blocks
14:52:03.042 INFO  Server - IPC Server listener on 0: starting
14:52:03.043 INFO  Server - IPC Server Responder: starting
14:52:03.044 INFO  BlockManager - Total number of blocks            = 0
14:52:03.044 INFO  BlockManager - Number of invalid blocks          = 0
14:52:03.044 INFO  BlockManager - Number of under-replicated blocks = 0
14:52:03.044 INFO  BlockManager - Number of  over-replicated blocks = 0
14:52:03.044 INFO  BlockManager - Number of blocks being written    = 0
14:52:03.044 INFO  StateChange - STATE* Replication Queue initialization scan for invalid, over- and under-replicated blocks completed in 4 msec
14:52:03.045 INFO  NameNode - NameNode RPC up at: localhost/127.0.0.1:42535
14:52:03.045 WARN  MetricsLoggerTask - Metrics logging will not be async since the logger is not log4j
14:52:03.045 INFO  FSNamesystem - Starting services required for active state
14:52:03.045 INFO  FSDirectory - Initializing quota with 12 thread(s)
14:52:03.045 INFO  FSDirectory - Quota initialization completed in 0 milliseconds
name space=1
storage space=0
storage types=RAM_DISK=0, SSD=0, DISK=0, ARCHIVE=0, PROVIDED=0
14:52:03.046 INFO  CacheReplicationMonitor - Starting CacheReplicationMonitor with interval 30000 milliseconds
14:52:03.047 INFO  MiniDFSCluster - Starting DataNode 0 with dfs.datanode.data.dir: [DISK]file:/tmp/minicluster_storage1617096220754514403/data/data1,[DISK]file:/tmp/minicluster_storage1617096220754514403/data/data2
14:52:03.048 INFO  ThrottledAsyncChecker - Scheduling a check for [DISK]file:/tmp/minicluster_storage1617096220754514403/data/data1
14:52:03.048 INFO  ThrottledAsyncChecker - Scheduling a check for [DISK]file:/tmp/minicluster_storage1617096220754514403/data/data2
14:52:03.059 INFO  MetricsSystemImpl - DataNode metrics system started (again)
14:52:03.059 INFO  Util - dfs.datanode.fileio.profiling.sampling.percentage set to 0. Disabling file IO profiling
14:52:03.059 INFO  BlockScanner - Initialized block scanner with targetBytesPerSec 1048576
14:52:03.060 INFO  DataNode - Configured hostname is 127.0.0.1
14:52:03.060 INFO  Util - dfs.datanode.fileio.profiling.sampling.percentage set to 0. Disabling file IO profiling
14:52:03.060 INFO  DataNode - Starting DataNode with maxLockedMemory = 0
14:52:03.060 INFO  DataNode - Opened streaming server at /127.0.0.1:40051
14:52:03.060 INFO  DataNode - Balancing bandwidth is 104857600 bytes/s
14:52:03.060 INFO  DataNode - Number threads for balancing is 100
14:52:03.062 WARN  AuthenticationFilter - Unable to initialize FileSignerSecretProvider, falling back to use random secrets. Reason: Could not read signature secret file: /home/runner/hadoop-http-auth-signature-secret
14:52:03.063 WARN  HttpRequestLog - Jetty request log can only be enabled using Log4j
14:52:03.064 INFO  HttpServer2 - Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)
14:52:03.065 INFO  HttpServer2 - Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context datanode
14:52:03.065 INFO  HttpServer2 - Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context static
14:52:03.065 INFO  HttpServer2 - Jetty bound to port 40741
14:52:03.065 INFO  Server - jetty-9.4.56.v20240826; built: 2024-08-26T17:15:05.868Z; git: ec6782ff5ead824dabdcf47fa98f90a4aedff401; jvm 17.0.6+10
14:52:03.066 INFO  session - DefaultSessionIdManager workerName=node0
14:52:03.066 INFO  session - No SessionScavenger set, using defaults
14:52:03.066 INFO  session - node0 Scavenging every 660000ms
14:52:03.066 INFO  ContextHandler - Started o.e.j.s.ServletContextHandler@7f4c0d4b{static,/static,jar:file:/home/runner/.gradle/caches/modules-2/files-2.1/org.apache.hadoop/hadoop-hdfs/3.3.6/5058b645375c6a68f509e167ad6a6ada9642df09/hadoop-hdfs-3.3.6-tests.jar!/webapps/static,AVAILABLE}
14:52:03.153 INFO  ContextHandler - Started o.e.j.w.WebAppContext@651c7ecc{datanode,/,file:///tmp/jetty-localhost-40741-hadoop-hdfs-3_3_6-tests_jar-_-any-1076438268305441842/webapp/,AVAILABLE}{jar:file:/home/runner/.gradle/caches/modules-2/files-2.1/org.apache.hadoop/hadoop-hdfs/3.3.6/5058b645375c6a68f509e167ad6a6ada9642df09/hadoop-hdfs-3.3.6-tests.jar!/webapps/datanode}
14:52:03.153 INFO  AbstractConnector - Started ServerConnector@205c88ed{HTTP/1.1, (http/1.1)}{localhost:40741}
14:52:03.153 INFO  Server - Started @82520ms
14:52:03.154 WARN  DatanodeHttpServer - Got null for restCsrfPreventionFilter - will not do any filtering.
14:52:03.155 INFO  DatanodeHttpServer - Listening HTTP traffic on /127.0.0.1:46403
14:52:03.155 INFO  DataNode - dnUserName = runner
14:52:03.155 INFO  DataNode - supergroup = supergroup
14:52:03.156 INFO  JvmPauseMonitor - Starting JVM pause monitor
14:52:03.156 INFO  CallQueueManager - Using callQueue: class java.util.concurrent.LinkedBlockingQueue, queueCapacity: 1000, scheduler: class org.apache.hadoop.ipc.DefaultRpcScheduler, ipcBackoff: false.
14:52:03.156 INFO  Server - Listener at localhost:40771
14:52:03.156 INFO  Server - Starting Socket Reader #1 for port 0
14:52:03.158 INFO  DataNode - Opened IPC server at /127.0.0.1:40771
14:52:03.168 INFO  DataNode - Refresh request received for nameservices: null
14:52:03.169 INFO  DataNode - Starting BPOfferServices for nameservices: <default>
14:52:03.169 WARN  MetricsLoggerTask - Metrics logging will not be async since the logger is not log4j
14:52:03.169 INFO  DataNode - Block pool <registering> (Datanode Uuid unassigned) service to localhost/127.0.0.1:42535 starting to offer service
14:52:03.170 INFO  Server - IPC Server Responder: starting
14:52:03.170 INFO  Server - IPC Server listener on 0: starting
14:52:03.171 INFO  DataNode - Acknowledging ACTIVE Namenode during handshakeBlock pool <registering> (Datanode Uuid unassigned) service to localhost/127.0.0.1:42535
14:52:03.171 INFO  Storage - Using 2 threads to upgrade data directories (dfs.datanode.parallel.volumes.load.threads.num=2, dataDirs=2)
14:52:03.172 INFO  Storage - Lock on /tmp/minicluster_storage1617096220754514403/data/data1/in_use.lock acquired by nodename 3348@runnervmnay03
14:52:03.172 INFO  Storage - Storage directory with location [DISK]file:/tmp/minicluster_storage1617096220754514403/data/data1 is not formatted for namespace 1622665296. Formatting...
14:52:03.173 INFO  Storage - Generated new storageID DS-cafd446e-26f3-4631-890c-f89194c6e843 for directory /tmp/minicluster_storage1617096220754514403/data/data1 
14:52:03.173 INFO  audit - allowed=true	ugi=runner (auth:SIMPLE)	ip=/127.0.0.1	cmd=datanodeReport	src=null	dst=null	perm=null	proto=rpc
14:52:03.174 INFO  MiniDFSCluster - dnInfo.length != numDataNodes
14:52:03.174 INFO  MiniDFSCluster - Waiting for cluster to become active
14:52:03.174 INFO  Storage - Lock on /tmp/minicluster_storage1617096220754514403/data/data2/in_use.lock acquired by nodename 3348@runnervmnay03
14:52:03.174 INFO  Storage - Storage directory with location [DISK]file:/tmp/minicluster_storage1617096220754514403/data/data2 is not formatted for namespace 1622665296. Formatting...
14:52:03.175 INFO  Storage - Generated new storageID DS-1353066f-501d-4d30-992a-903b44831000 for directory /tmp/minicluster_storage1617096220754514403/data/data2 
14:52:03.189 INFO  Storage - Analyzing storage directories for bpid BP-1183507661-10.1.0.125-1772635922860
14:52:03.190 INFO  Storage - Locking is disabled for /tmp/minicluster_storage1617096220754514403/data/data1/current/BP-1183507661-10.1.0.125-1772635922860
14:52:03.190 INFO  Storage - Block pool storage directory for location [DISK]file:/tmp/minicluster_storage1617096220754514403/data/data1 and block pool id BP-1183507661-10.1.0.125-1772635922860 is not formatted. Formatting ...
14:52:03.190 INFO  Storage - Formatting block pool BP-1183507661-10.1.0.125-1772635922860 directory /tmp/minicluster_storage1617096220754514403/data/data1/current/BP-1183507661-10.1.0.125-1772635922860/current
14:52:03.206 INFO  Storage - Analyzing storage directories for bpid BP-1183507661-10.1.0.125-1772635922860
14:52:03.206 INFO  Storage - Locking is disabled for /tmp/minicluster_storage1617096220754514403/data/data2/current/BP-1183507661-10.1.0.125-1772635922860
14:52:03.207 INFO  Storage - Block pool storage directory for location [DISK]file:/tmp/minicluster_storage1617096220754514403/data/data2 and block pool id BP-1183507661-10.1.0.125-1772635922860 is not formatted. Formatting ...
14:52:03.207 INFO  Storage - Formatting block pool BP-1183507661-10.1.0.125-1772635922860 directory /tmp/minicluster_storage1617096220754514403/data/data2/current/BP-1183507661-10.1.0.125-1772635922860/current
14:52:03.208 INFO  DataNode - Setting up storage: nsid=1622665296;bpid=BP-1183507661-10.1.0.125-1772635922860;lv=-57;nsInfo=lv=-66;cid=testClusterID;nsid=1622665296;c=1772635922860;bpid=BP-1183507661-10.1.0.125-1772635922860;dnuuid=null
14:52:03.209 INFO  DataNode - Generated and persisted new Datanode UUID c4241e56-ba2e-4cff-b78f-7cc0c44245ed
14:52:03.209 INFO  FsDatasetImpl - The datanode lock is a read write lock
14:52:03.210 INFO  FsDatasetImpl - Added new volume: DS-cafd446e-26f3-4631-890c-f89194c6e843
14:52:03.210 INFO  FsDatasetImpl - Added volume - [DISK]file:/tmp/minicluster_storage1617096220754514403/data/data1, StorageType: DISK
14:52:03.211 INFO  FsDatasetImpl - Added new volume: DS-1353066f-501d-4d30-992a-903b44831000
14:52:03.211 INFO  FsDatasetImpl - Added volume - [DISK]file:/tmp/minicluster_storage1617096220754514403/data/data2, StorageType: DISK
14:52:03.211 INFO  MemoryMappableBlockLoader - Initializing cache loader: MemoryMappableBlockLoader.
14:52:03.211 INFO  FsDatasetImpl - Registered FSDatasetState MBean
14:52:03.212 INFO  FsDatasetImpl - Adding block pool BP-1183507661-10.1.0.125-1772635922860
14:52:03.212 INFO  FsDatasetImpl - Scanning block pool BP-1183507661-10.1.0.125-1772635922860 on volume /tmp/minicluster_storage1617096220754514403/data/data1...
14:52:03.212 WARN  FsDatasetImpl - dfsUsed file missing in /tmp/minicluster_storage1617096220754514403/data/data1/current/BP-1183507661-10.1.0.125-1772635922860/current, will proceed with Du for space computation calculation, 
14:52:03.213 INFO  FsDatasetImpl - Scanning block pool BP-1183507661-10.1.0.125-1772635922860 on volume /tmp/minicluster_storage1617096220754514403/data/data2...
14:52:03.213 WARN  FsDatasetImpl - dfsUsed file missing in /tmp/minicluster_storage1617096220754514403/data/data2/current/BP-1183507661-10.1.0.125-1772635922860/current, will proceed with Du for space computation calculation, 
14:52:03.228 INFO  FsDatasetImpl - Time taken to scan block pool BP-1183507661-10.1.0.125-1772635922860 on /tmp/minicluster_storage1617096220754514403/data/data2: 15ms
14:52:03.232 INFO  FsDatasetImpl - Time taken to scan block pool BP-1183507661-10.1.0.125-1772635922860 on /tmp/minicluster_storage1617096220754514403/data/data1: 20ms
14:52:03.232 INFO  FsDatasetImpl - Total time to scan all replicas for block pool BP-1183507661-10.1.0.125-1772635922860: 20ms
14:52:03.232 INFO  FsDatasetImpl - Adding replicas to map for block pool BP-1183507661-10.1.0.125-1772635922860 on volume /tmp/minicluster_storage1617096220754514403/data/data1...
14:52:03.232 INFO  BlockPoolSlice - Replica Cache file: /tmp/minicluster_storage1617096220754514403/data/data1/current/BP-1183507661-10.1.0.125-1772635922860/current/replicas doesn't exist 
14:52:03.233 INFO  FsDatasetImpl - Adding replicas to map for block pool BP-1183507661-10.1.0.125-1772635922860 on volume /tmp/minicluster_storage1617096220754514403/data/data2...
14:52:03.233 INFO  BlockPoolSlice - Replica Cache file: /tmp/minicluster_storage1617096220754514403/data/data2/current/BP-1183507661-10.1.0.125-1772635922860/current/replicas doesn't exist 
14:52:03.233 INFO  FsDatasetImpl - Time to add replicas to map for block pool BP-1183507661-10.1.0.125-1772635922860 on volume /tmp/minicluster_storage1617096220754514403/data/data2: 0ms
14:52:03.233 INFO  FsDatasetImpl - Time to add replicas to map for block pool BP-1183507661-10.1.0.125-1772635922860 on volume /tmp/minicluster_storage1617096220754514403/data/data1: 0ms
14:52:03.233 INFO  FsDatasetImpl - Total time to add all replicas to map for block pool BP-1183507661-10.1.0.125-1772635922860: 1ms
14:52:03.233 INFO  ThrottledAsyncChecker - Scheduling a check for /tmp/minicluster_storage1617096220754514403/data/data1
14:52:03.233 INFO  DatasetVolumeChecker - Scheduled health check for volume /tmp/minicluster_storage1617096220754514403/data/data1
14:52:03.233 INFO  ThrottledAsyncChecker - Scheduling a check for /tmp/minicluster_storage1617096220754514403/data/data2
14:52:03.233 INFO  DatasetVolumeChecker - Scheduled health check for volume /tmp/minicluster_storage1617096220754514403/data/data2
14:52:03.234 INFO  VolumeScanner - Now scanning bpid BP-1183507661-10.1.0.125-1772635922860 on volume /tmp/minicluster_storage1617096220754514403/data/data2
14:52:03.234 WARN  DirectoryScanner - dfs.datanode.directoryscan.throttle.limit.ms.per.sec set to value above 1000 ms/sec. Assuming default value of -1
14:52:03.234 INFO  DirectoryScanner - Periodic Directory Tree Verification scan starting in 15090190ms with interval of 21600000ms and throttle limit of -1ms/s
14:52:03.234 INFO  VolumeScanner - VolumeScanner(/tmp/minicluster_storage1617096220754514403/data/data2, DS-1353066f-501d-4d30-992a-903b44831000): finished scanning block pool BP-1183507661-10.1.0.125-1772635922860
14:52:03.234 INFO  VolumeScanner - Now scanning bpid BP-1183507661-10.1.0.125-1772635922860 on volume /tmp/minicluster_storage1617096220754514403/data/data1
14:52:03.234 INFO  DataNode - Block pool BP-1183507661-10.1.0.125-1772635922860 (Datanode Uuid c4241e56-ba2e-4cff-b78f-7cc0c44245ed) service to localhost/127.0.0.1:42535 beginning handshake with NN
14:52:03.234 INFO  VolumeScanner - VolumeScanner(/tmp/minicluster_storage1617096220754514403/data/data2, DS-1353066f-501d-4d30-992a-903b44831000): no suitable block pools found to scan.  Waiting 1814400000 ms.
14:52:03.234 INFO  VolumeScanner - VolumeScanner(/tmp/minicluster_storage1617096220754514403/data/data1, DS-cafd446e-26f3-4631-890c-f89194c6e843): finished scanning block pool BP-1183507661-10.1.0.125-1772635922860
14:52:03.235 INFO  VolumeScanner - VolumeScanner(/tmp/minicluster_storage1617096220754514403/data/data1, DS-cafd446e-26f3-4631-890c-f89194c6e843): no suitable block pools found to scan.  Waiting 1814399999 ms.
14:52:03.235 INFO  StateChange - BLOCK* registerDatanode: from DatanodeRegistration(127.0.0.1:40051, datanodeUuid=c4241e56-ba2e-4cff-b78f-7cc0c44245ed, infoPort=46403, infoSecurePort=0, ipcPort=40771, storageInfo=lv=-57;cid=testClusterID;nsid=1622665296;c=1772635922860) storage c4241e56-ba2e-4cff-b78f-7cc0c44245ed
14:52:03.235 INFO  NetworkTopology - Adding a new node: /default-rack/127.0.0.1:40051
14:52:03.235 INFO  BlockReportLeaseManager - Registered DN c4241e56-ba2e-4cff-b78f-7cc0c44245ed (127.0.0.1:40051).
14:52:03.235 INFO  DataNode - Block pool BP-1183507661-10.1.0.125-1772635922860 (Datanode Uuid c4241e56-ba2e-4cff-b78f-7cc0c44245ed) service to localhost/127.0.0.1:42535 successfully registered with NN
14:52:03.235 INFO  DataNode - For namenode localhost/127.0.0.1:42535 using BLOCKREPORT_INTERVAL of 21600000msecs CACHEREPORT_INTERVAL of 10000msecs Initial delay: 0msecs; heartBeatInterval=3000
14:52:03.236 INFO  DataNode - Starting IBR Task Handler.
14:52:03.236 INFO  DatanodeDescriptor - Adding new storage ID DS-cafd446e-26f3-4631-890c-f89194c6e843 for DN 127.0.0.1:40051
14:52:03.236 INFO  DatanodeDescriptor - Adding new storage ID DS-1353066f-501d-4d30-992a-903b44831000 for DN 127.0.0.1:40051
14:52:03.236 INFO  DataNode - After receiving heartbeat response, updating state of namenode localhost:42535 to active
14:52:03.237 INFO  BlockStateChange - BLOCK* processReport 0x761ee03e05650f9e with lease ID 0xc538171f5d0dd41d: Processing first storage report for DS-1353066f-501d-4d30-992a-903b44831000 from datanode DatanodeRegistration(127.0.0.1:40051, datanodeUuid=c4241e56-ba2e-4cff-b78f-7cc0c44245ed, infoPort=46403, infoSecurePort=0, ipcPort=40771, storageInfo=lv=-57;cid=testClusterID;nsid=1622665296;c=1772635922860)
14:52:03.237 INFO  BlockStateChange - BLOCK* processReport 0x761ee03e05650f9e with lease ID 0xc538171f5d0dd41d: from storage DS-1353066f-501d-4d30-992a-903b44831000 node DatanodeRegistration(127.0.0.1:40051, datanodeUuid=c4241e56-ba2e-4cff-b78f-7cc0c44245ed, infoPort=46403, infoSecurePort=0, ipcPort=40771, storageInfo=lv=-57;cid=testClusterID;nsid=1622665296;c=1772635922860), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0
14:52:03.237 INFO  BlockStateChange - BLOCK* processReport 0x761ee03e05650f9e with lease ID 0xc538171f5d0dd41d: Processing first storage report for DS-cafd446e-26f3-4631-890c-f89194c6e843 from datanode DatanodeRegistration(127.0.0.1:40051, datanodeUuid=c4241e56-ba2e-4cff-b78f-7cc0c44245ed, infoPort=46403, infoSecurePort=0, ipcPort=40771, storageInfo=lv=-57;cid=testClusterID;nsid=1622665296;c=1772635922860)
14:52:03.237 INFO  BlockStateChange - BLOCK* processReport 0x761ee03e05650f9e with lease ID 0xc538171f5d0dd41d: from storage DS-cafd446e-26f3-4631-890c-f89194c6e843 node DatanodeRegistration(127.0.0.1:40051, datanodeUuid=c4241e56-ba2e-4cff-b78f-7cc0c44245ed, infoPort=46403, infoSecurePort=0, ipcPort=40771, storageInfo=lv=-57;cid=testClusterID;nsid=1622665296;c=1772635922860), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0
14:52:03.237 INFO  DataNode - Successfully sent block report 0x761ee03e05650f9e with lease ID 0xc538171f5d0dd41d to namenode: localhost/127.0.0.1:42535,  containing 2 storage report(s), of which we sent 2. The reports had 0 total blocks and used 1 RPC(s). This took 0 msecs to generate and 2 msecs for RPC and NN processing. Got back one command: FinalizeCommand/5.
14:52:03.238 INFO  DataNode - Got finalize command for block pool BP-1183507661-10.1.0.125-1772635922860
14:52:03.274 INFO  audit - allowed=true	ugi=runner (auth:SIMPLE)	ip=/127.0.0.1	cmd=datanodeReport	src=null	dst=null	perm=null	proto=rpc
14:52:03.275 INFO  MiniDFSCluster - Cluster is active
14:52:03.284 INFO  MemoryStore - Block broadcast_616 stored as values in memory (estimated size 3.0 KiB, free 1919.1 MiB)
14:52:03.285 INFO  MemoryStore - Block broadcast_616_piece0 stored as bytes in memory (estimated size 525.0 B, free 1919.1 MiB)
14:52:03.285 INFO  BlockManagerInfo - Added broadcast_616_piece0 in memory on localhost:44923 (size: 525.0 B, free: 1919.8 MiB)
14:52:03.286 INFO  SparkContext - Created broadcast 616 from broadcast at VcfSink.java:65
14:52:03.290 INFO  PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
14:52:03.290 INFO  FileOutputCommitter - File Output Committer Algorithm version is 1
14:52:03.290 INFO  FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
14:52:03.312 INFO  SparkContext - Starting job: runJob at SparkHadoopWriter.scala:83
14:52:03.312 INFO  DAGScheduler - Registering RDD 1485 (mapToPair at VariantsSparkSink.java:140) as input to shuffle 53
14:52:03.313 INFO  DAGScheduler - Got job 228 (runJob at SparkHadoopWriter.scala:83) with 1 output partitions
14:52:03.313 INFO  DAGScheduler - Final stage: ResultStage 293 (runJob at SparkHadoopWriter.scala:83)
14:52:03.313 INFO  DAGScheduler - Parents of final stage: List(ShuffleMapStage 292)
14:52:03.313 INFO  DAGScheduler - Missing parents: List(ShuffleMapStage 292)
14:52:03.313 INFO  DAGScheduler - Submitting ShuffleMapStage 292 (MapPartitionsRDD[1485] at mapToPair at VariantsSparkSink.java:140), which has no missing parents
14:52:03.315 INFO  MemoryStore - Block broadcast_617 stored as values in memory (estimated size 6.5 KiB, free 1919.1 MiB)
14:52:03.315 INFO  MemoryStore - Block broadcast_617_piece0 stored as bytes in memory (estimated size 3.7 KiB, free 1919.1 MiB)
14:52:03.315 INFO  BlockManagerInfo - Added broadcast_617_piece0 in memory on localhost:44923 (size: 3.7 KiB, free: 1919.8 MiB)
14:52:03.315 INFO  SparkContext - Created broadcast 617 from broadcast at DAGScheduler.scala:1580
14:52:03.316 INFO  DAGScheduler - Submitting 4 missing tasks from ShuffleMapStage 292 (MapPartitionsRDD[1485] at mapToPair at VariantsSparkSink.java:140) (first 15 tasks are for partitions Vector(0, 1, 2, 3))
14:52:03.316 INFO  TaskSchedulerImpl - Adding task set 292.0 with 4 tasks resource profile 0
14:52:03.317 INFO  TaskSetManager - Starting task 0.0 in stage 292.0 (TID 379) (localhost, executor driver, partition 0, PROCESS_LOCAL, 8087 bytes) 
14:52:03.318 INFO  TaskSetManager - Starting task 1.0 in stage 292.0 (TID 380) (localhost, executor driver, partition 1, PROCESS_LOCAL, 8193 bytes) 
14:52:03.318 INFO  TaskSetManager - Starting task 2.0 in stage 292.0 (TID 381) (localhost, executor driver, partition 2, PROCESS_LOCAL, 8087 bytes) 
14:52:03.318 INFO  TaskSetManager - Starting task 3.0 in stage 292.0 (TID 382) (localhost, executor driver, partition 3, PROCESS_LOCAL, 8193 bytes) 
14:52:03.319 INFO  Executor - Running task 3.0 in stage 292.0 (TID 382)
14:52:03.319 INFO  Executor - Running task 0.0 in stage 292.0 (TID 379)
14:52:03.319 INFO  Executor - Running task 1.0 in stage 292.0 (TID 380)
14:52:03.319 INFO  Executor - Running task 2.0 in stage 292.0 (TID 381)
14:52:03.336 INFO  Executor - Finished task 1.0 in stage 292.0 (TID 380). 1062 bytes result sent to driver
14:52:03.336 INFO  TaskSetManager - Finished task 1.0 in stage 292.0 (TID 380) in 19 ms on localhost (executor driver) (1/4)
14:52:03.336 INFO  Executor - Finished task 3.0 in stage 292.0 (TID 382). 1062 bytes result sent to driver
14:52:03.337 INFO  Executor - Finished task 0.0 in stage 292.0 (TID 379). 1062 bytes result sent to driver
14:52:03.337 INFO  Executor - Finished task 2.0 in stage 292.0 (TID 381). 1062 bytes result sent to driver
14:52:03.337 INFO  TaskSetManager - Finished task 3.0 in stage 292.0 (TID 382) in 19 ms on localhost (executor driver) (2/4)
14:52:03.337 INFO  TaskSetManager - Finished task 2.0 in stage 292.0 (TID 381) in 19 ms on localhost (executor driver) (3/4)
14:52:03.337 INFO  TaskSetManager - Finished task 0.0 in stage 292.0 (TID 379) in 21 ms on localhost (executor driver) (4/4)
14:52:03.338 INFO  TaskSchedulerImpl - Removed TaskSet 292.0, whose tasks have all completed, from pool 
14:52:03.338 INFO  DAGScheduler - ShuffleMapStage 292 (mapToPair at VariantsSparkSink.java:140) finished in 0.025 s
14:52:03.338 INFO  DAGScheduler - looking for newly runnable stages
14:52:03.338 INFO  DAGScheduler - running: HashSet()
14:52:03.338 INFO  DAGScheduler - waiting: HashSet(ResultStage 293)
14:52:03.338 INFO  DAGScheduler - failed: HashSet()
14:52:03.338 INFO  DAGScheduler - Submitting ResultStage 293 (MapPartitionsRDD[1489] at mapToPair at VcfSink.java:77), which has no missing parents
14:52:03.346 INFO  MemoryStore - Block broadcast_618 stored as values in memory (estimated size 149.6 KiB, free 1918.9 MiB)
14:52:03.346 INFO  MemoryStore - Block broadcast_618_piece0 stored as bytes in memory (estimated size 56.2 KiB, free 1918.9 MiB)
14:52:03.347 INFO  BlockManagerInfo - Added broadcast_618_piece0 in memory on localhost:44923 (size: 56.2 KiB, free: 1919.7 MiB)
14:52:03.347 INFO  SparkContext - Created broadcast 618 from broadcast at DAGScheduler.scala:1580
14:52:03.347 INFO  DAGScheduler - Submitting 1 missing tasks from ResultStage 293 (MapPartitionsRDD[1489] at mapToPair at VcfSink.java:77) (first 15 tasks are for partitions Vector(0))
14:52:03.347 INFO  TaskSchedulerImpl - Adding task set 293.0 with 1 tasks resource profile 0
14:52:03.348 INFO  TaskSetManager - Starting task 0.0 in stage 293.0 (TID 383) (localhost, executor driver, partition 0, NODE_LOCAL, 7513 bytes) 
14:52:03.348 INFO  Executor - Running task 0.0 in stage 293.0 (TID 383)
14:52:03.355 INFO  ShuffleBlockFetcherIterator - Getting 4 (1336.0 B) non-empty blocks including 4 (1336.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
14:52:03.355 INFO  ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
14:52:03.362 INFO  PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
14:52:03.362 INFO  FileOutputCommitter - File Output Committer Algorithm version is 1
14:52:03.362 INFO  FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
14:52:03.362 INFO  PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
14:52:03.362 INFO  FileOutputCommitter - File Output Committer Algorithm version is 1
14:52:03.362 INFO  FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
14:52:03.379 INFO  FileOutputCommitter - Saved output of task 'attempt_202603041452032394824293291967586_1489_r_000000_0' to file:/tmp/VariantsSparkSinkUnitTest15246426870432333601.vcf.parts/_temporary/0/task_202603041452032394824293291967586_1489_r_000000
14:52:03.379 INFO  SparkHadoopMapRedUtil - attempt_202603041452032394824293291967586_1489_r_000000_0: Committed. Elapsed time: 0 ms.
14:52:03.379 INFO  Executor - Finished task 0.0 in stage 293.0 (TID 383). 1858 bytes result sent to driver
14:52:03.380 INFO  TaskSetManager - Finished task 0.0 in stage 293.0 (TID 383) in 32 ms on localhost (executor driver) (1/1)
14:52:03.380 INFO  TaskSchedulerImpl - Removed TaskSet 293.0, whose tasks have all completed, from pool 
14:52:03.380 INFO  DAGScheduler - ResultStage 293 (runJob at SparkHadoopWriter.scala:83) finished in 0.042 s
14:52:03.380 INFO  DAGScheduler - Job 228 is finished. Cancelling potential speculative or zombie tasks for this job
14:52:03.380 INFO  TaskSchedulerImpl - Killing all running tasks in stage 293: Stage finished
14:52:03.380 INFO  DAGScheduler - Job 228 finished: runJob at SparkHadoopWriter.scala:83, took 0.068448 s
14:52:03.381 INFO  SparkHadoopWriter - Start to commit write Job job_202603041452032394824293291967586_1489.
14:52:03.386 INFO  SparkHadoopWriter - Write Job job_202603041452032394824293291967586_1489 committed. Elapsed time: 5 ms.
14:52:03.396 INFO  HadoopFileSystemWrapper - Concatenating 2 parts to /tmp/VariantsSparkSinkUnitTest15246426870432333601.vcf
14:52:03.401 INFO  HadoopFileSystemWrapper - Concatenating to /tmp/VariantsSparkSinkUnitTest15246426870432333601.vcf done
[March 4, 2026 at 2:52:03 PM UTC] org.broadinstitute.hellbender.tools.IndexFeatureFile done. Elapsed time: 0.00 minutes.
Runtime.totalMemory()=1396703232
[March 4, 2026 at 2:52:03 PM UTC] org.broadinstitute.hellbender.tools.IndexFeatureFile done. Elapsed time: 0.00 minutes.
Runtime.totalMemory()=1396703232
[March 4, 2026 at 2:52:03 PM UTC] org.broadinstitute.hellbender.tools.IndexFeatureFile done. Elapsed time: 0.00 minutes.
Runtime.totalMemory()=1396703232
[March 4, 2026 at 2:52:04 PM UTC] org.broadinstitute.hellbender.tools.IndexFeatureFile done. Elapsed time: 0.00 minutes.
Runtime.totalMemory()=1396703232