17:37:43.966 INFO BlockManagerInfo - Removed broadcast_20_piece0 on localhost:37953 in memory (size: 2.4 KiB, free: 1920.0 MiB)
17:37:43.969 INFO BlockManagerInfo - Removed broadcast_30_piece0 on localhost:37953 in memory (size: 4.7 KiB, free: 1920.0 MiB)
17:37:43.971 INFO BlockManagerInfo - Removed broadcast_27_piece0 on localhost:37953 in memory (size: 5.1 KiB, free: 1920.0 MiB)
17:37:43.976 INFO BlockManagerInfo - Removed broadcast_24_piece0 on localhost:37953 in memory (size: 4.3 KiB, free: 1920.0 MiB)
17:37:43.978 INFO BlockManagerInfo - Removed broadcast_19_piece0 on localhost:37953 in memory (size: 2.4 KiB, free: 1920.0 MiB)
17:37:43.980 INFO BlockManagerInfo - Removed broadcast_25_piece0 on localhost:37953 in memory (size: 4.5 KiB, free: 1920.0 MiB)
17:37:43.985 INFO MiniDFSCluster - starting cluster: numNameNodes=1, numDataNodes=1
17:37:44.304 INFO NameNode - Formatting using clusterid: testClusterID
17:37:44.315 INFO FSEditLog - Edit logging is async:true
17:37:44.330 INFO FSNamesystem - KeyProvider: null
17:37:44.331 INFO FSNamesystem - fsLock is fair: true
17:37:44.331 INFO FSNamesystem - Detailed lock hold time metrics enabled: false
17:37:44.331 INFO FSNamesystem - fsOwner = runner (auth:SIMPLE)
17:37:44.331 INFO FSNamesystem - supergroup = supergroup
17:37:44.331 INFO FSNamesystem - isPermissionEnabled = true
17:37:44.331 INFO FSNamesystem - isStoragePolicyEnabled = true
17:37:44.331 INFO FSNamesystem - HA Enabled: false
17:37:44.363 INFO Util - dfs.datanode.fileio.profiling.sampling.percentage set to 0. Disabling file IO profiling
17:37:44.367 INFO deprecation - hadoop.configured.node.mapping is deprecated. Instead, use net.topology.configured.node.mapping
17:37:44.367 INFO DatanodeManager - dfs.block.invalidate.limit : configured=1000, counted=60, effected=1000
17:37:44.367 INFO DatanodeManager - dfs.namenode.datanode.registration.ip-hostname-check=true
17:37:44.369 INFO BlockManager - dfs.namenode.startup.delay.block.deletion.sec is set to 000:00:00:00.000
17:37:44.370 INFO BlockManager - The block deletion will start around 2025 Oct 27 17:37:44
17:37:44.371 INFO GSet - Computing capacity for map BlocksMap
17:37:44.371 INFO GSet - VM type = 64-bit
17:37:44.371 INFO GSet - 2.0% max memory 3.4 GB = 70 MB
17:37:44.371 INFO GSet - capacity = 2^23 = 8388608 entries
17:37:44.376 INFO BlockManager - Storage policy satisfier is disabled
17:37:44.376 INFO BlockManager - dfs.block.access.token.enable = false
17:37:44.381 INFO BlockManagerSafeMode - dfs.namenode.safemode.threshold-pct = 0.999
17:37:44.381 INFO BlockManagerSafeMode - dfs.namenode.safemode.min.datanodes = 0
17:37:44.381 INFO BlockManagerSafeMode - dfs.namenode.safemode.extension = 0
17:37:44.381 INFO BlockManager - defaultReplication = 1
17:37:44.381 INFO BlockManager - maxReplication = 512
17:37:44.381 INFO BlockManager - minReplication = 1
17:37:44.381 INFO BlockManager - maxReplicationStreams = 2
17:37:44.381 INFO BlockManager - redundancyRecheckInterval = 3000ms
17:37:44.381 INFO BlockManager - encryptDataTransfer = false
17:37:44.381 INFO BlockManager - maxNumBlocksToLog = 1000
17:37:44.400 INFO FSDirectory - GLOBAL serial map: bits=29 maxEntries=536870911
17:37:44.400 INFO FSDirectory - USER serial map: bits=24 maxEntries=16777215
17:37:44.400 INFO FSDirectory - GROUP serial map: bits=24 maxEntries=16777215
17:37:44.400 INFO FSDirectory - XATTR serial map: bits=24 maxEntries=16777215
17:37:44.407 INFO GSet - Computing capacity for map INodeMap
17:37:44.407 INFO GSet - VM type = 64-bit
17:37:44.407 INFO GSet - 1.0% max memory 3.4 GB = 35 MB
17:37:44.407 INFO GSet - capacity = 2^22 = 4194304 entries
17:37:44.409 INFO FSDirectory - ACLs enabled? true
17:37:44.409 INFO FSDirectory - POSIX ACL inheritance enabled? true
17:37:44.409 INFO FSDirectory - XAttrs enabled? true
17:37:44.409 INFO NameNode - Caching file names occurring more than 10 times
17:37:44.412 INFO SnapshotManager - Loaded config captureOpenFiles: false, skipCaptureAccessTimeOnlyChange: false, snapshotDiffAllowSnapRootDescendant: true, maxSnapshotLimit: 65536
17:37:44.413 INFO SnapshotManager - SkipList is disabled
17:37:44.416 INFO GSet - Computing capacity for map cachedBlocks
17:37:44.416 INFO GSet - VM type = 64-bit
17:37:44.416 INFO GSet - 0.25% max memory 3.4 GB = 8.8 MB
17:37:44.416 INFO GSet - capacity = 2^20 = 1048576 entries
17:37:44.421 INFO TopMetrics - NNTop conf: dfs.namenode.top.window.num.buckets = 10
17:37:44.421 INFO TopMetrics - NNTop conf: dfs.namenode.top.num.users = 10
17:37:44.421 INFO TopMetrics - NNTop conf: dfs.namenode.top.windows.minutes = 1,5,25
17:37:44.423 INFO FSNamesystem - Retry cache on namenode is enabled
17:37:44.423 INFO FSNamesystem - Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis
17:37:44.424 INFO GSet - Computing capacity for map NameNodeRetryCache
17:37:44.424 INFO GSet - VM type = 64-bit
17:37:44.424 INFO GSet - 0.029999999329447746% max memory 3.4 GB = 1.0 MB
17:37:44.424 INFO GSet - capacity = 2^17 = 131072 entries
17:37:44.435 INFO FSImage - Allocated new BlockPoolId: BP-617366194-10.1.0.252-1761586664431
17:37:44.450 INFO Storage - Storage directory /tmp/minicluster_storage14473397491642437130/name-0-1 has been successfully formatted.
17:37:44.457 INFO Storage - Storage directory /tmp/minicluster_storage14473397491642437130/name-0-2 has been successfully formatted.
17:37:44.470 INFO FSImageFormatProtobuf - Saving image file /tmp/minicluster_storage14473397491642437130/name-0-2/current/fsimage.ckpt_0000000000000000000 using no compression
17:37:44.470 INFO FSImageFormatProtobuf - Saving image file /tmp/minicluster_storage14473397491642437130/name-0-1/current/fsimage.ckpt_0000000000000000000 using no compression
17:37:44.530 INFO FSImageFormatProtobuf - Image file /tmp/minicluster_storage14473397491642437130/name-0-1/current/fsimage.ckpt_0000000000000000000 of size 401 bytes saved in 0 seconds .
17:37:44.530 INFO FSImageFormatProtobuf - Image file /tmp/minicluster_storage14473397491642437130/name-0-2/current/fsimage.ckpt_0000000000000000000 of size 401 bytes saved in 0 seconds .
17:37:44.547 INFO NNStorageRetentionManager - Going to retain 1 images with txid >= 0
17:37:44.551 INFO FSNamesystem - Stopping services started for active state
17:37:44.551 INFO FSNamesystem - Stopping services started for standby state
17:37:44.552 INFO NameNode - createNameNode []
17:37:44.592 WARN MetricsConfig - Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
17:37:44.601 INFO MetricsSystemImpl - Scheduled Metric snapshot period at 10 second(s).
17:37:44.601 INFO MetricsSystemImpl - NameNode metrics system started
17:37:44.604 INFO NameNodeUtils - fs.defaultFS is hdfs://127.0.0.1:0
17:37:44.627 INFO JvmPauseMonitor - Starting JVM pause monitor
17:37:44.638 INFO DFSUtil - Filter initializers set : org.apache.hadoop.http.lib.StaticUserWebFilter,org.apache.hadoop.hdfs.web.AuthFilterInitializer
17:37:44.643 INFO DFSUtil - Starting Web-server for hdfs at: http://localhost:0
17:37:44.652 INFO log - Logging initialized @26763ms to org.eclipse.jetty.util.log.Slf4jLog
17:37:44.723 WARN AuthenticationFilter - Unable to initialize FileSignerSecretProvider, falling back to use random secrets. Reason: Could not read signature secret file: /home/runner/hadoop-http-auth-signature-secret
17:37:44.726 WARN HttpRequestLog - Jetty request log can only be enabled using Log4j
17:37:44.729 INFO HttpServer2 - Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)
17:37:44.730 INFO HttpServer2 - Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context hdfs
17:37:44.730 INFO HttpServer2 - Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context static
17:37:44.732 INFO HttpServer2 - Added filter AuthFilter (class=org.apache.hadoop.hdfs.web.AuthFilter) to context hdfs
17:37:44.732 INFO HttpServer2 - Added filter AuthFilter (class=org.apache.hadoop.hdfs.web.AuthFilter) to context static
17:37:44.758 INFO HttpServer2 - addJerseyResourcePackage: packageName=org.apache.hadoop.hdfs.server.namenode.web.resources;org.apache.hadoop.hdfs.web.resources, pathSpec=/webhdfs/v1/*
17:37:44.761 INFO HttpServer2 - Jetty bound to port 36035
17:37:44.762 INFO Server - jetty-9.4.56.v20240826; built: 2024-08-26T17:15:05.868Z; git: ec6782ff5ead824dabdcf47fa98f90a4aedff401; jvm 17.0.6+10
17:37:44.783 INFO session - DefaultSessionIdManager workerName=node0
17:37:44.783 INFO session - No SessionScavenger set, using defaults
17:37:44.784 INFO session - node0 Scavenging every 660000ms
17:37:44.795 WARN AuthenticationFilter - Unable to initialize FileSignerSecretProvider, falling back to use random secrets. Reason: Could not read signature secret file: /home/runner/hadoop-http-auth-signature-secret
17:37:44.797 INFO ContextHandler - Started o.e.j.s.ServletContextHandler@8381edd{static,/static,jar:file:/home/runner/.gradle/caches/modules-2/files-2.1/org.apache.hadoop/hadoop-hdfs/3.3.6/5058b645375c6a68f509e167ad6a6ada9642df09/hadoop-hdfs-3.3.6-tests.jar!/webapps/static,AVAILABLE}
17:37:44.909 INFO ContextHandler - Started o.e.j.w.WebAppContext@371c7b39{hdfs,/,file:///tmp/jetty-localhost-36035-hadoop-hdfs-3_3_6-tests_jar-_-any-4925090116424057802/webapp/,AVAILABLE}{jar:file:/home/runner/.gradle/caches/modules-2/files-2.1/org.apache.hadoop/hadoop-hdfs/3.3.6/5058b645375c6a68f509e167ad6a6ada9642df09/hadoop-hdfs-3.3.6-tests.jar!/webapps/hdfs}
17:37:44.914 INFO AbstractConnector - Started ServerConnector@1d0d4478{HTTP/1.1, (http/1.1)}{localhost:36035}
17:37:44.914 INFO Server - Started @27026ms
17:37:44.919 INFO FSEditLog - Edit logging is async:true
17:37:44.927 INFO FSNamesystem - KeyProvider: null
17:37:44.927 INFO FSNamesystem - fsLock is fair: true
17:37:44.927 INFO FSNamesystem - Detailed lock hold time metrics enabled: false
17:37:44.927 INFO FSNamesystem - fsOwner = runner (auth:SIMPLE)
17:37:44.927 INFO FSNamesystem - supergroup = supergroup
17:37:44.927 INFO FSNamesystem - isPermissionEnabled = true
17:37:44.927 INFO FSNamesystem - isStoragePolicyEnabled = true
17:37:44.927 INFO FSNamesystem - HA Enabled: false
17:37:44.928 INFO Util - dfs.datanode.fileio.profiling.sampling.percentage set to 0. Disabling file IO profiling
17:37:44.928 INFO DatanodeManager - dfs.block.invalidate.limit : configured=1000, counted=60, effected=1000
17:37:44.928 INFO DatanodeManager - dfs.namenode.datanode.registration.ip-hostname-check=true
17:37:44.928 INFO BlockManager - dfs.namenode.startup.delay.block.deletion.sec is set to 000:00:00:00.000
17:37:44.928 INFO BlockManager - The block deletion will start around 2025 Oct 27 17:37:44
17:37:44.928 INFO GSet - Computing capacity for map BlocksMap
17:37:44.928 INFO GSet - VM type = 64-bit
17:37:44.928 INFO GSet - 2.0% max memory 3.4 GB = 70 MB
17:37:44.928 INFO GSet - capacity = 2^23 = 8388608 entries
17:37:44.930 INFO BlockManager - Storage policy satisfier is disabled
17:37:44.930 INFO BlockManager - dfs.block.access.token.enable = false
17:37:44.930 INFO BlockManagerSafeMode - dfs.namenode.safemode.threshold-pct = 0.999
17:37:44.930 INFO BlockManagerSafeMode - dfs.namenode.safemode.min.datanodes = 0
17:37:44.930 INFO BlockManagerSafeMode - dfs.namenode.safemode.extension = 0
17:37:44.930 INFO BlockManager - defaultReplication = 1
17:37:44.930 INFO BlockManager - maxReplication = 512
17:37:44.930 INFO BlockManager - minReplication = 1
17:37:44.930 INFO BlockManager - maxReplicationStreams = 2
17:37:44.930 INFO BlockManager - redundancyRecheckInterval = 3000ms
17:37:44.930 INFO BlockManager - encryptDataTransfer = false
17:37:44.930 INFO BlockManager - maxNumBlocksToLog = 1000
17:37:44.930 INFO GSet - Computing capacity for map INodeMap
17:37:44.930 INFO GSet - VM type = 64-bit
17:37:44.930 INFO GSet - 1.0% max memory 3.4 GB = 35 MB
17:37:44.930 INFO GSet - capacity = 2^22 = 4194304 entries
17:37:44.931 INFO FSDirectory - ACLs enabled? true
17:37:44.931 INFO FSDirectory - POSIX ACL inheritance enabled? true
17:37:44.931 INFO FSDirectory - XAttrs enabled? true
17:37:44.931 INFO NameNode - Caching file names occurring more than 10 times
17:37:44.931 INFO SnapshotManager - Loaded config captureOpenFiles: false, skipCaptureAccessTimeOnlyChange: false, snapshotDiffAllowSnapRootDescendant: true, maxSnapshotLimit: 65536
17:37:44.931 INFO SnapshotManager - SkipList is disabled
17:37:44.931 INFO GSet - Computing capacity for map cachedBlocks
17:37:44.931 INFO GSet - VM type = 64-bit
17:37:44.931 INFO GSet - 0.25% max memory 3.4 GB = 8.8 MB
17:37:44.932 INFO GSet - capacity = 2^20 = 1048576 entries
17:37:44.932 INFO TopMetrics - NNTop conf: dfs.namenode.top.window.num.buckets = 10
17:37:44.932 INFO TopMetrics - NNTop conf: dfs.namenode.top.num.users = 10
17:37:44.932 INFO TopMetrics - NNTop conf: dfs.namenode.top.windows.minutes = 1,5,25
17:37:44.932 INFO FSNamesystem - Retry cache on namenode is enabled
17:37:44.932 INFO FSNamesystem - Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis
17:37:44.932 INFO GSet - Computing capacity for map NameNodeRetryCache
17:37:44.932 INFO GSet - VM type = 64-bit
17:37:44.932 INFO GSet - 0.029999999329447746% max memory 3.4 GB = 1.0 MB
17:37:44.932 INFO GSet - capacity = 2^17 = 131072 entries
17:37:44.939 INFO Storage - Lock on /tmp/minicluster_storage14473397491642437130/name-0-1/in_use.lock acquired by nodename 2975@runnervmrcw8b
17:37:44.944 INFO Storage - Lock on /tmp/minicluster_storage14473397491642437130/name-0-2/in_use.lock acquired by nodename 2975@runnervmrcw8b
17:37:44.945 INFO FileJournalManager - Recovering unfinalized segments in /tmp/minicluster_storage14473397491642437130/name-0-1/current
17:37:44.945 INFO FileJournalManager - Recovering unfinalized segments in /tmp/minicluster_storage14473397491642437130/name-0-2/current
17:37:44.946 INFO FSImage - No edit log streams selected.
17:37:44.946 INFO FSImage - Planning to load image: FSImageFile(file=/tmp/minicluster_storage14473397491642437130/name-0-1/current/fsimage_0000000000000000000, cpktTxId=0000000000000000000)
17:37:44.962 INFO FSImageFormatPBINode - Loading 1 INodes.
17:37:44.962 INFO FSImageFormatPBINode - Successfully loaded 1 inodes
17:37:44.965 INFO FSImageFormatPBINode - Completed update blocks map and name cache, total waiting duration 0ms.
17:37:44.966 INFO FSImageFormatProtobuf - Loaded FSImage in 0 seconds.
17:37:44.966 INFO FSImage - Loaded image for txid 0 from /tmp/minicluster_storage14473397491642437130/name-0-1/current/fsimage_0000000000000000000
17:37:44.969 INFO FSNamesystem - Need to save fs image? false (staleImage=false, haEnabled=false, isRollingUpgrade=false)
17:37:44.969 INFO FSEditLog - Starting log segment at 1
17:37:44.984 INFO NameCache - initialized with 0 entries 0 lookups
17:37:44.984 INFO FSNamesystem - Finished loading FSImage in 50 msecs
17:37:45.055 INFO NameNode - RPC server is binding to localhost:0
17:37:45.055 INFO NameNode - Enable NameNode state context:false
17:37:45.058 INFO CallQueueManager - Using callQueue: class java.util.concurrent.LinkedBlockingQueue, queueCapacity: 1000, scheduler: class org.apache.hadoop.ipc.DefaultRpcScheduler, ipcBackoff: false.
17:37:45.066 INFO Server - Listener at localhost:38243
17:37:45.066 INFO Server - Starting Socket Reader #1 for port 0
17:37:45.089 INFO NameNode - Clients are to use localhost:38243 to access this namenode/service.
17:37:45.091 INFO FSNamesystem - Registered FSNamesystemState, ReplicatedBlocksState and ECBlockGroupsState MBeans.
17:37:45.116 INFO LeaseManager - Number of blocks under construction: 0
17:37:45.120 INFO DatanodeAdminDefaultMonitor - Initialized the Default Decommission and Maintenance monitor
17:37:45.121 INFO BlockManager - Start MarkedDeleteBlockScrubber thread
17:37:45.122 INFO BlockManager - initializing replication queues
17:37:45.122 INFO StateChange - STATE* Leaving safe mode after 0 secs
17:37:45.122 INFO StateChange - STATE* Network topology has 0 racks and 0 datanodes
17:37:45.122 INFO StateChange - STATE* UnderReplicatedBlocks has 0 blocks
17:37:45.125 INFO BlockManager - Total number of blocks = 0
17:37:45.125 INFO BlockManager - Number of invalid blocks = 0
17:37:45.125 INFO BlockManager - Number of under-replicated blocks = 0
17:37:45.125 INFO BlockManager - Number of over-replicated blocks = 0
17:37:45.125 INFO BlockManager - Number of blocks being written = 0
17:37:45.125 INFO StateChange - STATE* Replication Queue initialization scan for invalid, over- and under-replicated blocks completed in 3 msec
17:37:45.135 INFO Server - IPC Server Responder: starting
17:37:45.136 INFO Server - IPC Server listener on 0: starting
17:37:45.138 INFO NameNode - NameNode RPC up at: localhost/127.0.0.1:38243
17:37:45.138 WARN MetricsLoggerTask - Metrics logging will not be async since the logger is not log4j
17:37:45.139 INFO FSNamesystem - Starting services required for active state
17:37:45.139 INFO FSDirectory - Initializing quota with 12 thread(s)
17:37:45.141 INFO FSDirectory - Quota initialization completed in 2 milliseconds
name space=1
storage space=0
storage types=RAM_DISK=0, SSD=0, DISK=0, ARCHIVE=0, PROVIDED=0
17:37:45.143 INFO CacheReplicationMonitor - Starting CacheReplicationMonitor with interval 30000 milliseconds
17:37:45.149 INFO MiniDFSCluster - Starting DataNode 0 with dfs.datanode.data.dir: [DISK]file:/tmp/minicluster_storage14473397491642437130/data/data1,[DISK]file:/tmp/minicluster_storage14473397491642437130/data/data2
17:37:45.159 INFO ThrottledAsyncChecker - Scheduling a check for [DISK]file:/tmp/minicluster_storage14473397491642437130/data/data1
17:37:45.166 INFO ThrottledAsyncChecker - Scheduling a check for [DISK]file:/tmp/minicluster_storage14473397491642437130/data/data2
17:37:45.174 INFO MetricsSystemImpl - DataNode metrics system started (again)
17:37:45.179 INFO Util - dfs.datanode.fileio.profiling.sampling.percentage set to 0. Disabling file IO profiling
17:37:45.181 INFO BlockScanner - Initialized block scanner with targetBytesPerSec 1048576
17:37:45.183 INFO DataNode - Configured hostname is 127.0.0.1
17:37:45.184 INFO Util - dfs.datanode.fileio.profiling.sampling.percentage set to 0. Disabling file IO profiling
17:37:45.184 INFO DataNode - Starting DataNode with maxLockedMemory = 0
17:37:45.187 INFO DataNode - Opened streaming server at /127.0.0.1:37927
17:37:45.188 INFO DataNode - Balancing bandwidth is 104857600 bytes/s
17:37:45.188 INFO DataNode - Number threads for balancing is 100
17:37:45.192 WARN AuthenticationFilter - Unable to initialize FileSignerSecretProvider, falling back to use random secrets. Reason: Could not read signature secret file: /home/runner/hadoop-http-auth-signature-secret
17:37:45.192 WARN HttpRequestLog - Jetty request log can only be enabled using Log4j
17:37:45.193 INFO HttpServer2 - Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)
17:37:45.193 INFO HttpServer2 - Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context datanode
17:37:45.193 INFO HttpServer2 - Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context static
17:37:45.195 INFO HttpServer2 - Jetty bound to port 40341
17:37:45.195 INFO Server - jetty-9.4.56.v20240826; built: 2024-08-26T17:15:05.868Z; git: ec6782ff5ead824dabdcf47fa98f90a4aedff401; jvm 17.0.6+10
17:37:45.195 INFO session - DefaultSessionIdManager workerName=node0
17:37:45.195 INFO session - No SessionScavenger set, using defaults
17:37:45.195 INFO session - node0 Scavenging every 600000ms
17:37:45.196 INFO ContextHandler - Started o.e.j.s.ServletContextHandler@10faf070{static,/static,jar:file:/home/runner/.gradle/caches/modules-2/files-2.1/org.apache.hadoop/hadoop-hdfs/3.3.6/5058b645375c6a68f509e167ad6a6ada9642df09/hadoop-hdfs-3.3.6-tests.jar!/webapps/static,AVAILABLE}
17:37:45.267 INFO ContextHandler - Started o.e.j.w.WebAppContext@7488db31{datanode,/,file:///tmp/jetty-localhost-40341-hadoop-hdfs-3_3_6-tests_jar-_-any-11718105098498108015/webapp/,AVAILABLE}{jar:file:/home/runner/.gradle/caches/modules-2/files-2.1/org.apache.hadoop/hadoop-hdfs/3.3.6/5058b645375c6a68f509e167ad6a6ada9642df09/hadoop-hdfs-3.3.6-tests.jar!/webapps/datanode}
17:37:45.267 INFO AbstractConnector - Started ServerConnector@57d53da4{HTTP/1.1, (http/1.1)}{localhost:40341}
17:37:45.268 INFO Server - Started @27379ms
17:37:45.272 WARN DatanodeHttpServer - Got null for restCsrfPreventionFilter - will not do any filtering.
17:37:45.273 INFO DatanodeHttpServer - Listening HTTP traffic on /127.0.0.1:36359
17:37:45.273 INFO JvmPauseMonitor - Starting JVM pause monitor
17:37:45.274 INFO DataNode - dnUserName = runner
17:37:45.274 INFO DataNode - supergroup = supergroup
17:37:45.282 INFO CallQueueManager - Using callQueue: class java.util.concurrent.LinkedBlockingQueue, queueCapacity: 1000, scheduler: class org.apache.hadoop.ipc.DefaultRpcScheduler, ipcBackoff: false.
17:37:45.282 INFO Server - Listener at localhost:38451
17:37:45.283 INFO Server - Starting Socket Reader #1 for port 0
17:37:45.286 INFO DataNode - Opened IPC server at /127.0.0.1:38451
17:37:45.299 INFO DataNode - Refresh request received for nameservices: null
17:37:45.300 INFO DataNode - Starting BPOfferServices for nameservices: <default>
17:37:45.307 INFO DataNode - Block pool <registering> (Datanode Uuid unassigned) service to localhost/127.0.0.1:38243 starting to offer service
17:37:45.309 WARN MetricsLoggerTask - Metrics logging will not be async since the logger is not log4j
17:37:45.310 INFO Server - IPC Server Responder: starting
17:37:45.310 INFO Server - IPC Server listener on 0: starting
17:37:45.410 INFO DataNode - Acknowledging ACTIVE Namenode during handshakeBlock pool <registering> (Datanode Uuid unassigned) service to localhost/127.0.0.1:38243
17:37:45.411 INFO Storage - Using 2 threads to upgrade data directories (dfs.datanode.parallel.volumes.load.threads.num=2, dataDirs=2)
17:37:45.416 INFO Storage - Lock on /tmp/minicluster_storage14473397491642437130/data/data1/in_use.lock acquired by nodename 2975@runnervmrcw8b
17:37:45.417 INFO Storage - Storage directory with location [DISK]file:/tmp/minicluster_storage14473397491642437130/data/data1 is not formatted for namespace 1487967217. Formatting...
17:37:45.418 INFO Storage - Generated new storageID DS-55e05c7c-458b-4ee5-8514-921ccedb9f70 for directory /tmp/minicluster_storage14473397491642437130/data/data1
17:37:45.423 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=datanodeReport src=null dst=null perm=null proto=rpc
17:37:45.427 INFO MiniDFSCluster - dnInfo.length != numDataNodes
17:37:45.427 INFO MiniDFSCluster - Waiting for cluster to become active
17:37:45.431 INFO Storage - Lock on /tmp/minicluster_storage14473397491642437130/data/data2/in_use.lock acquired by nodename 2975@runnervmrcw8b
17:37:45.431 INFO Storage - Storage directory with location [DISK]file:/tmp/minicluster_storage14473397491642437130/data/data2 is not formatted for namespace 1487967217. Formatting...
17:37:45.431 INFO Storage - Generated new storageID DS-e0d71477-3be5-4c55-b3b5-8e68aba28769 for directory /tmp/minicluster_storage14473397491642437130/data/data2
17:37:45.450 INFO Storage - Analyzing storage directories for bpid BP-617366194-10.1.0.252-1761586664431
17:37:45.450 INFO Storage - Locking is disabled for /tmp/minicluster_storage14473397491642437130/data/data1/current/BP-617366194-10.1.0.252-1761586664431
17:37:45.451 INFO Storage - Block pool storage directory for location [DISK]file:/tmp/minicluster_storage14473397491642437130/data/data1 and block pool id BP-617366194-10.1.0.252-1761586664431 is not formatted. Formatting ...
17:37:45.451 INFO Storage - Formatting block pool BP-617366194-10.1.0.252-1761586664431 directory /tmp/minicluster_storage14473397491642437130/data/data1/current/BP-617366194-10.1.0.252-1761586664431/current
17:37:45.468 INFO Storage - Analyzing storage directories for bpid BP-617366194-10.1.0.252-1761586664431
17:37:45.468 INFO Storage - Locking is disabled for /tmp/minicluster_storage14473397491642437130/data/data2/current/BP-617366194-10.1.0.252-1761586664431
17:37:45.468 INFO Storage - Block pool storage directory for location [DISK]file:/tmp/minicluster_storage14473397491642437130/data/data2 and block pool id BP-617366194-10.1.0.252-1761586664431 is not formatted. Formatting ...
17:37:45.468 INFO Storage - Formatting block pool BP-617366194-10.1.0.252-1761586664431 directory /tmp/minicluster_storage14473397491642437130/data/data2/current/BP-617366194-10.1.0.252-1761586664431/current
17:37:45.475 INFO DataNode - Setting up storage: nsid=1487967217;bpid=BP-617366194-10.1.0.252-1761586664431;lv=-57;nsInfo=lv=-66;cid=testClusterID;nsid=1487967217;c=1761586664431;bpid=BP-617366194-10.1.0.252-1761586664431;dnuuid=null
17:37:45.488 INFO DataNode - Generated and persisted new Datanode UUID 379dd5e5-59c3-42cd-b9d3-67ca5e44560b
17:37:45.497 INFO FsDatasetImpl - The datanode lock is a read write lock
17:37:45.524 INFO FsDatasetImpl - Added new volume: DS-55e05c7c-458b-4ee5-8514-921ccedb9f70
17:37:45.524 INFO FsDatasetImpl - Added volume - [DISK]file:/tmp/minicluster_storage14473397491642437130/data/data1, StorageType: DISK
17:37:45.525 INFO FsDatasetImpl - Added new volume: DS-e0d71477-3be5-4c55-b3b5-8e68aba28769
17:37:45.525 INFO FsDatasetImpl - Added volume - [DISK]file:/tmp/minicluster_storage14473397491642437130/data/data2, StorageType: DISK
17:37:45.527 INFO MemoryMappableBlockLoader - Initializing cache loader: MemoryMappableBlockLoader.
17:37:45.528 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=datanodeReport src=null dst=null perm=null proto=rpc
17:37:45.528 INFO MiniDFSCluster - dnInfo.length != numDataNodes
17:37:45.528 INFO MiniDFSCluster - Waiting for cluster to become active
17:37:45.539 INFO FsDatasetImpl - Registered FSDatasetState MBean
17:37:45.542 INFO FsDatasetImpl - Adding block pool BP-617366194-10.1.0.252-1761586664431
17:37:45.543 INFO BlockManagerInfo - Removed broadcast_23_piece0 on localhost:37953 in memory (size: 465.0 B, free: 1920.0 MiB)
17:37:45.543 INFO FsDatasetImpl - Scanning block pool BP-617366194-10.1.0.252-1761586664431 on volume /tmp/minicluster_storage14473397491642437130/data/data1...
17:37:45.544 INFO FsDatasetImpl - Scanning block pool BP-617366194-10.1.0.252-1761586664431 on volume /tmp/minicluster_storage14473397491642437130/data/data2...
17:37:45.547 INFO BlockManagerInfo - Removed broadcast_22_piece0 on localhost:37953 in memory (size: 159.0 B, free: 1920.0 MiB)
17:37:45.548 WARN FsDatasetImpl - dfsUsed file missing in /tmp/minicluster_storage14473397491642437130/data/data1/current/BP-617366194-10.1.0.252-1761586664431/current, will proceed with Du for space computation calculation,
17:37:45.548 WARN FsDatasetImpl - dfsUsed file missing in /tmp/minicluster_storage14473397491642437130/data/data2/current/BP-617366194-10.1.0.252-1761586664431/current, will proceed with Du for space computation calculation,
17:37:45.551 INFO BlockManagerInfo - Removed broadcast_33_piece0 on localhost:37953 in memory (size: 4.8 KiB, free: 1920.0 MiB)
17:37:45.554 INFO BlockManagerInfo - Removed broadcast_28_piece0 on localhost:37953 in memory (size: 320.0 B, free: 1920.0 MiB)
17:37:45.559 INFO BlockManager - Removing RDD 47
17:37:45.563 INFO BlockManagerInfo - Removed broadcast_31_piece0 on localhost:37953 in memory (size: 320.0 B, free: 1920.0 MiB)
17:37:45.564 INFO FsDatasetImpl - Time taken to scan block pool BP-617366194-10.1.0.252-1761586664431 on /tmp/minicluster_storage14473397491642437130/data/data1: 22ms
17:37:45.564 INFO FsDatasetImpl - Time taken to scan block pool BP-617366194-10.1.0.252-1761586664431 on /tmp/minicluster_storage14473397491642437130/data/data2: 21ms
17:37:45.565 INFO FsDatasetImpl - Total time to scan all replicas for block pool BP-617366194-10.1.0.252-1761586664431: 23ms
17:37:45.568 INFO FsDatasetImpl - Adding replicas to map for block pool BP-617366194-10.1.0.252-1761586664431 on volume /tmp/minicluster_storage14473397491642437130/data/data1...
17:37:45.568 INFO BlockPoolSlice - Replica Cache file: /tmp/minicluster_storage14473397491642437130/data/data1/current/BP-617366194-10.1.0.252-1761586664431/current/replicas doesn't exist
17:37:45.569 INFO FsDatasetImpl - Adding replicas to map for block pool BP-617366194-10.1.0.252-1761586664431 on volume /tmp/minicluster_storage14473397491642437130/data/data2...
17:37:45.569 INFO BlockPoolSlice - Replica Cache file: /tmp/minicluster_storage14473397491642437130/data/data2/current/BP-617366194-10.1.0.252-1761586664431/current/replicas doesn't exist
17:37:45.569 INFO FsDatasetImpl - Time to add replicas to map for block pool BP-617366194-10.1.0.252-1761586664431 on volume /tmp/minicluster_storage14473397491642437130/data/data1: 1ms
17:37:45.570 INFO FsDatasetImpl - Time to add replicas to map for block pool BP-617366194-10.1.0.252-1761586664431 on volume /tmp/minicluster_storage14473397491642437130/data/data2: 1ms
17:37:45.570 INFO FsDatasetImpl - Total time to add all replicas to map for block pool BP-617366194-10.1.0.252-1761586664431: 6ms
17:37:45.571 INFO ThrottledAsyncChecker - Scheduling a check for /tmp/minicluster_storage14473397491642437130/data/data1
17:37:45.575 INFO DatasetVolumeChecker - Scheduled health check for volume /tmp/minicluster_storage14473397491642437130/data/data1
17:37:45.576 INFO ThrottledAsyncChecker - Scheduling a check for /tmp/minicluster_storage14473397491642437130/data/data2
17:37:45.576 INFO DatasetVolumeChecker - Scheduled health check for volume /tmp/minicluster_storage14473397491642437130/data/data2
17:37:45.577 INFO VolumeScanner - Now scanning bpid BP-617366194-10.1.0.252-1761586664431 on volume /tmp/minicluster_storage14473397491642437130/data/data1
17:37:45.577 INFO VolumeScanner - Now scanning bpid BP-617366194-10.1.0.252-1761586664431 on volume /tmp/minicluster_storage14473397491642437130/data/data2
17:37:45.578 INFO VolumeScanner - VolumeScanner(/tmp/minicluster_storage14473397491642437130/data/data2, DS-e0d71477-3be5-4c55-b3b5-8e68aba28769): finished scanning block pool BP-617366194-10.1.0.252-1761586664431
17:37:45.578 INFO VolumeScanner - VolumeScanner(/tmp/minicluster_storage14473397491642437130/data/data1, DS-55e05c7c-458b-4ee5-8514-921ccedb9f70): finished scanning block pool BP-617366194-10.1.0.252-1761586664431
17:37:45.579 WARN DirectoryScanner - dfs.datanode.directoryscan.throttle.limit.ms.per.sec set to value above 1000 ms/sec. Assuming default value of -1
17:37:45.579 INFO DirectoryScanner - Periodic Directory Tree Verification scan starting in 4931329ms with interval of 21600000ms and throttle limit of -1ms/s
17:37:45.581 INFO VolumeScanner - VolumeScanner(/tmp/minicluster_storage14473397491642437130/data/data1, DS-55e05c7c-458b-4ee5-8514-921ccedb9f70): no suitable block pools found to scan. Waiting 1814399996 ms.
17:37:45.581 INFO VolumeScanner - VolumeScanner(/tmp/minicluster_storage14473397491642437130/data/data2, DS-e0d71477-3be5-4c55-b3b5-8e68aba28769): no suitable block pools found to scan. Waiting 1814399996 ms.
17:37:45.583 INFO DataNode - Block pool BP-617366194-10.1.0.252-1761586664431 (Datanode Uuid 379dd5e5-59c3-42cd-b9d3-67ca5e44560b) service to localhost/127.0.0.1:38243 beginning handshake with NN
17:37:45.592 INFO StateChange - BLOCK* registerDatanode: from DatanodeRegistration(127.0.0.1:37927, datanodeUuid=379dd5e5-59c3-42cd-b9d3-67ca5e44560b, infoPort=36359, infoSecurePort=0, ipcPort=38451, storageInfo=lv=-57;cid=testClusterID;nsid=1487967217;c=1761586664431) storage 379dd5e5-59c3-42cd-b9d3-67ca5e44560b
17:37:45.593 INFO NetworkTopology - Adding a new node: /default-rack/127.0.0.1:37927
17:37:45.593 INFO BlockReportLeaseManager - Registered DN 379dd5e5-59c3-42cd-b9d3-67ca5e44560b (127.0.0.1:37927).
17:37:45.597 INFO DataNode - Block pool BP-617366194-10.1.0.252-1761586664431 (Datanode Uuid 379dd5e5-59c3-42cd-b9d3-67ca5e44560b) service to localhost/127.0.0.1:38243 successfully registered with NN
17:37:45.597 INFO DataNode - For namenode localhost/127.0.0.1:38243 using BLOCKREPORT_INTERVAL of 21600000msecs CACHEREPORT_INTERVAL of 10000msecs Initial delay: 0msecs; heartBeatInterval=3000
17:37:45.597 INFO DataNode - Starting IBR Task Handler.
17:37:45.605 INFO DatanodeDescriptor - Adding new storage ID DS-55e05c7c-458b-4ee5-8514-921ccedb9f70 for DN 127.0.0.1:37927
17:37:45.605 INFO DatanodeDescriptor - Adding new storage ID DS-e0d71477-3be5-4c55-b3b5-8e68aba28769 for DN 127.0.0.1:37927
17:37:45.610 INFO DataNode - After receiving heartbeat response, updating state of namenode localhost:38243 to active
17:37:45.622 INFO BlockStateChange - BLOCK* processReport 0xd5f96f043bcb8706 with lease ID 0x9b6a68ee500abb8d: Processing first storage report for DS-55e05c7c-458b-4ee5-8514-921ccedb9f70 from datanode DatanodeRegistration(127.0.0.1:37927, datanodeUuid=379dd5e5-59c3-42cd-b9d3-67ca5e44560b, infoPort=36359, infoSecurePort=0, ipcPort=38451, storageInfo=lv=-57;cid=testClusterID;nsid=1487967217;c=1761586664431)
17:37:45.623 INFO BlockStateChange - BLOCK* processReport 0xd5f96f043bcb8706 with lease ID 0x9b6a68ee500abb8d: from storage DS-55e05c7c-458b-4ee5-8514-921ccedb9f70 node DatanodeRegistration(127.0.0.1:37927, datanodeUuid=379dd5e5-59c3-42cd-b9d3-67ca5e44560b, infoPort=36359, infoSecurePort=0, ipcPort=38451, storageInfo=lv=-57;cid=testClusterID;nsid=1487967217;c=1761586664431), blocks: 0, hasStaleStorage: true, processing time: 1 msecs, invalidatedBlocks: 0
17:37:45.623 INFO BlockStateChange - BLOCK* processReport 0xd5f96f043bcb8706 with lease ID 0x9b6a68ee500abb8d: Processing first storage report for DS-e0d71477-3be5-4c55-b3b5-8e68aba28769 from datanode DatanodeRegistration(127.0.0.1:37927, datanodeUuid=379dd5e5-59c3-42cd-b9d3-67ca5e44560b, infoPort=36359, infoSecurePort=0, ipcPort=38451, storageInfo=lv=-57;cid=testClusterID;nsid=1487967217;c=1761586664431)
17:37:45.624 INFO BlockStateChange - BLOCK* processReport 0xd5f96f043bcb8706 with lease ID 0x9b6a68ee500abb8d: from storage DS-e0d71477-3be5-4c55-b3b5-8e68aba28769 node DatanodeRegistration(127.0.0.1:37927, datanodeUuid=379dd5e5-59c3-42cd-b9d3-67ca5e44560b, infoPort=36359, infoSecurePort=0, ipcPort=38451, storageInfo=lv=-57;cid=testClusterID;nsid=1487967217;c=1761586664431), blocks: 0, hasStaleStorage: false, processing time: 1 msecs, invalidatedBlocks: 0
17:37:45.630 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=datanodeReport src=null dst=null perm=null proto=rpc
17:37:45.632 INFO DataNode - Successfully sent block report 0xd5f96f043bcb8706 with lease ID 0x9b6a68ee500abb8d to namenode: localhost/127.0.0.1:38243, containing 2 storage report(s), of which we sent 2. The reports had 0 total blocks and used 1 RPC(s). This took 1 msecs to generate and 20 msecs for RPC and NN processing. Got back one command: FinalizeCommand/5.
17:37:45.632 INFO DataNode - Got finalize command for block pool BP-617366194-10.1.0.252-1761586664431
17:37:45.634 INFO MiniDFSCluster - Cluster is active
17:37:45.665 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:37:45.684 INFO MemoryStore - Block broadcast_34 stored as values in memory (estimated size 297.9 KiB, free 1919.7 MiB)
17:37:45.700 INFO MemoryStore - Block broadcast_34_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1919.7 MiB)
17:37:45.700 INFO BlockManagerInfo - Added broadcast_34_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1920.0 MiB)
17:37:45.700 INFO SparkContext - Created broadcast 34 from newAPIHadoopFile at PathSplitSource.java:96
17:37:45.751 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:37:45.755 INFO MemoryStore - Block broadcast_35 stored as values in memory (estimated size 297.9 KiB, free 1919.4 MiB)
17:37:45.766 INFO MemoryStore - Block broadcast_35_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1919.3 MiB)
17:37:45.766 INFO BlockManagerInfo - Added broadcast_35_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.9 MiB)
17:37:45.767 INFO SparkContext - Created broadcast 35 from newAPIHadoopFile at PathSplitSource.java:96
17:37:45.822 INFO FileInputFormat - Total input files to process : 1
17:37:45.834 INFO MemoryStore - Block broadcast_36 stored as values in memory (estimated size 160.7 KiB, free 1919.2 MiB)
17:37:45.838 INFO MemoryStore - Block broadcast_36_piece0 stored as bytes in memory (estimated size 9.6 KiB, free 1919.2 MiB)
17:37:45.839 INFO BlockManagerInfo - Added broadcast_36_piece0 in memory on localhost:37953 (size: 9.6 KiB, free: 1919.9 MiB)
17:37:45.839 INFO SparkContext - Created broadcast 36 from broadcast at ReadsSparkSink.java:133
17:37:45.848 INFO MemoryStore - Block broadcast_37 stored as values in memory (estimated size 163.2 KiB, free 1919.0 MiB)
17:37:45.851 INFO MemoryStore - Block broadcast_37_piece0 stored as bytes in memory (estimated size 9.6 KiB, free 1919.0 MiB)
17:37:45.851 INFO BlockManagerInfo - Added broadcast_37_piece0 in memory on localhost:37953 (size: 9.6 KiB, free: 1919.9 MiB)
17:37:45.852 INFO SparkContext - Created broadcast 37 from broadcast at BamSink.java:76
17:37:45.870 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam.parts dst=null perm=null proto=rpc
17:37:45.874 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:37:45.875 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:37:45.875 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:37:45.890 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=mkdirs src=/user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam.parts/_temporary/0 dst=null perm=runner:supergroup:rwxr-xr-x proto=rpc
17:37:45.902 INFO SparkContext - Starting job: runJob at SparkHadoopWriter.scala:83
17:37:45.903 INFO DAGScheduler - Registering RDD 77 (mapToPair at SparkUtils.java:161) as input to shuffle 7
17:37:45.903 INFO DAGScheduler - Got job 20 (runJob at SparkHadoopWriter.scala:83) with 1 output partitions
17:37:45.903 INFO DAGScheduler - Final stage: ResultStage 30 (runJob at SparkHadoopWriter.scala:83)
17:37:45.903 INFO DAGScheduler - Parents of final stage: List(ShuffleMapStage 29)
17:37:45.903 INFO DAGScheduler - Missing parents: List(ShuffleMapStage 29)
17:37:45.904 INFO DAGScheduler - Submitting ShuffleMapStage 29 (MapPartitionsRDD[77] at mapToPair at SparkUtils.java:161), which has no missing parents
17:37:45.933 INFO MemoryStore - Block broadcast_38 stored as values in memory (estimated size 520.4 KiB, free 1918.5 MiB)
17:37:45.935 INFO MemoryStore - Block broadcast_38_piece0 stored as bytes in memory (estimated size 166.1 KiB, free 1918.3 MiB)
17:37:45.935 INFO BlockManagerInfo - Added broadcast_38_piece0 in memory on localhost:37953 (size: 166.1 KiB, free: 1919.7 MiB)
17:37:45.935 INFO SparkContext - Created broadcast 38 from broadcast at DAGScheduler.scala:1580
17:37:45.935 INFO DAGScheduler - Submitting 1 missing tasks from ShuffleMapStage 29 (MapPartitionsRDD[77] at mapToPair at SparkUtils.java:161) (first 15 tasks are for partitions Vector(0))
17:37:45.935 INFO TaskSchedulerImpl - Adding task set 29.0 with 1 tasks resource profile 0
17:37:45.938 INFO TaskSetManager - Starting task 0.0 in stage 29.0 (TID 67) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7861 bytes)
17:37:45.939 INFO Executor - Running task 0.0 in stage 29.0 (TID 67)
17:37:46.008 INFO NewHadoopRDD - Input split: file:/home/runner/work/gatk/gatk/src/test/resources/org/broadinstitute/hellbender/tools/BQSR/HiSeq.1mb.1RG.2k_lines.bam:0+222075
17:37:46.014 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:37:46.016 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:37:46.035 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:37:46.055 INFO Executor - Finished task 0.0 in stage 29.0 (TID 67). 1148 bytes result sent to driver
17:37:46.056 INFO TaskSetManager - Finished task 0.0 in stage 29.0 (TID 67) in 120 ms on localhost (executor driver) (1/1)
17:37:46.056 INFO TaskSchedulerImpl - Removed TaskSet 29.0, whose tasks have all completed, from pool
17:37:46.056 INFO DAGScheduler - ShuffleMapStage 29 (mapToPair at SparkUtils.java:161) finished in 0.146 s
17:37:46.056 INFO DAGScheduler - looking for newly runnable stages
17:37:46.056 INFO DAGScheduler - running: HashSet()
17:37:46.056 INFO DAGScheduler - waiting: HashSet(ResultStage 30)
17:37:46.056 INFO DAGScheduler - failed: HashSet()
17:37:46.056 INFO DAGScheduler - Submitting ResultStage 30 (MapPartitionsRDD[82] at mapToPair at BamSink.java:91), which has no missing parents
17:37:46.064 INFO MemoryStore - Block broadcast_39 stored as values in memory (estimated size 241.5 KiB, free 1918.1 MiB)
17:37:46.065 INFO MemoryStore - Block broadcast_39_piece0 stored as bytes in memory (estimated size 67.1 KiB, free 1918.0 MiB)
17:37:46.065 INFO BlockManagerInfo - Added broadcast_39_piece0 in memory on localhost:37953 (size: 67.1 KiB, free: 1919.7 MiB)
17:37:46.065 INFO SparkContext - Created broadcast 39 from broadcast at DAGScheduler.scala:1580
17:37:46.065 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 30 (MapPartitionsRDD[82] at mapToPair at BamSink.java:91) (first 15 tasks are for partitions Vector(0))
17:37:46.065 INFO TaskSchedulerImpl - Adding task set 30.0 with 1 tasks resource profile 0
17:37:46.066 INFO TaskSetManager - Starting task 0.0 in stage 30.0 (TID 68) (localhost, executor driver, partition 0, NODE_LOCAL, 7513 bytes)
17:37:46.067 INFO Executor - Running task 0.0 in stage 30.0 (TID 68)
17:37:46.086 INFO ShuffleBlockFetcherIterator - Getting 1 (343.8 KiB) non-empty blocks including 1 (343.8 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
17:37:46.086 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
17:37:46.157 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:37:46.157 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:37:46.157 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:37:46.158 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:37:46.158 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:37:46.158 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:37:46.176 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam.parts/_temporary/0/_temporary/attempt_202510271737459220188902038722646_0082_r_000000_0/part-r-00000 dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:37:46.190 WARN IntelDeflaterFactory - Intel Deflater not supported, using Java.util.zip.Deflater
17:37:46.192 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam.parts/_temporary/0/_temporary/attempt_202510271737459220188902038722646_0082_r_000000_0/.part-r-00000.sbi dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:37:46.194 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam.parts/_temporary/0/_temporary/attempt_202510271737459220188902038722646_0082_r_000000_0/.part-r-00000.bai dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:37:46.213 INFO StateChange - BLOCK* allocate blk_1073741825_1001, replicas=127.0.0.1:37927 for /user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam.parts/_temporary/0/_temporary/attempt_202510271737459220188902038722646_0082_r_000000_0/part-r-00000
17:37:46.244 INFO DataNode - Receiving BP-617366194-10.1.0.252-1761586664431:blk_1073741825_1001 src: /127.0.0.1:38934 dest: /127.0.0.1:37927
17:37:46.267 INFO clienttrace - src: /127.0.0.1:38934, dest: /127.0.0.1:37927, bytes: 233040, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1369393814_1, offset: 0, srvID: 379dd5e5-59c3-42cd-b9d3-67ca5e44560b, blockid: BP-617366194-10.1.0.252-1761586664431:blk_1073741825_1001, duration(ns): 4710941
17:37:46.268 INFO DataNode - PacketResponder: BP-617366194-10.1.0.252-1761586664431:blk_1073741825_1001, type=LAST_IN_PIPELINE terminating
17:37:46.274 INFO FSNamesystem - BLOCK* blk_1073741825_1001 is COMMITTED but not COMPLETE(numNodes= 0 < minimum = 1) in file /user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam.parts/_temporary/0/_temporary/attempt_202510271737459220188902038722646_0082_r_000000_0/part-r-00000
17:37:46.677 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam.parts/_temporary/0/_temporary/attempt_202510271737459220188902038722646_0082_r_000000_0/part-r-00000 is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:37:46.678 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam.parts/_temporary/0/_temporary/attempt_202510271737459220188902038722646_0082_r_000000_0/part-r-00000 dst=null perm=null proto=rpc
17:37:46.681 INFO StateChange - BLOCK* allocate blk_1073741826_1002, replicas=127.0.0.1:37927 for /user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam.parts/_temporary/0/_temporary/attempt_202510271737459220188902038722646_0082_r_000000_0/.part-r-00000.sbi
17:37:46.682 INFO DataNode - Receiving BP-617366194-10.1.0.252-1761586664431:blk_1073741826_1002 src: /127.0.0.1:38944 dest: /127.0.0.1:37927
17:37:46.684 INFO clienttrace - src: /127.0.0.1:38944, dest: /127.0.0.1:37927, bytes: 212, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1369393814_1, offset: 0, srvID: 379dd5e5-59c3-42cd-b9d3-67ca5e44560b, blockid: BP-617366194-10.1.0.252-1761586664431:blk_1073741826_1002, duration(ns): 692376
17:37:46.684 INFO DataNode - PacketResponder: BP-617366194-10.1.0.252-1761586664431:blk_1073741826_1002, type=LAST_IN_PIPELINE terminating
17:37:46.685 INFO FSNamesystem - BLOCK* blk_1073741826_1002 is COMMITTED but not COMPLETE(numNodes= 0 < minimum = 1) in file /user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam.parts/_temporary/0/_temporary/attempt_202510271737459220188902038722646_0082_r_000000_0/.part-r-00000.sbi
17:37:47.087 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam.parts/_temporary/0/_temporary/attempt_202510271737459220188902038722646_0082_r_000000_0/.part-r-00000.sbi is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:37:47.090 INFO StateChange - BLOCK* allocate blk_1073741827_1003, replicas=127.0.0.1:37927 for /user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam.parts/_temporary/0/_temporary/attempt_202510271737459220188902038722646_0082_r_000000_0/.part-r-00000.bai
17:37:47.092 INFO DataNode - Receiving BP-617366194-10.1.0.252-1761586664431:blk_1073741827_1003 src: /127.0.0.1:38952 dest: /127.0.0.1:37927
17:37:47.093 INFO clienttrace - src: /127.0.0.1:38952, dest: /127.0.0.1:37927, bytes: 5472, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1369393814_1, offset: 0, srvID: 379dd5e5-59c3-42cd-b9d3-67ca5e44560b, blockid: BP-617366194-10.1.0.252-1761586664431:blk_1073741827_1003, duration(ns): 765137
17:37:47.093 INFO DataNode - PacketResponder: BP-617366194-10.1.0.252-1761586664431:blk_1073741827_1003, type=LAST_IN_PIPELINE terminating
17:37:47.094 INFO FSNamesystem - BLOCK* blk_1073741827_1003 is COMMITTED but not COMPLETE(numNodes= 0 < minimum = 1) in file /user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam.parts/_temporary/0/_temporary/attempt_202510271737459220188902038722646_0082_r_000000_0/.part-r-00000.bai
17:37:47.496 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam.parts/_temporary/0/_temporary/attempt_202510271737459220188902038722646_0082_r_000000_0/.part-r-00000.bai is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:37:47.499 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam.parts/_temporary/0/_temporary/attempt_202510271737459220188902038722646_0082_r_000000_0 dst=null perm=null proto=rpc
17:37:47.502 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam.parts/_temporary/0/_temporary/attempt_202510271737459220188902038722646_0082_r_000000_0 dst=null perm=null proto=rpc
17:37:47.503 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam.parts/_temporary/0/task_202510271737459220188902038722646_0082_r_000000 dst=null perm=null proto=rpc
17:37:47.509 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=rename src=/user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam.parts/_temporary/0/_temporary/attempt_202510271737459220188902038722646_0082_r_000000_0 dst=/user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam.parts/_temporary/0/task_202510271737459220188902038722646_0082_r_000000 perm=runner:supergroup:rwxr-xr-x proto=rpc
17:37:47.510 INFO FileOutputCommitter - Saved output of task 'attempt_202510271737459220188902038722646_0082_r_000000_0' to hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam.parts/_temporary/0/task_202510271737459220188902038722646_0082_r_000000
17:37:47.511 INFO SparkHadoopMapRedUtil - attempt_202510271737459220188902038722646_0082_r_000000_0: Committed. Elapsed time: 8 ms.
17:37:47.512 INFO Executor - Finished task 0.0 in stage 30.0 (TID 68). 1858 bytes result sent to driver
17:37:47.514 INFO TaskSetManager - Finished task 0.0 in stage 30.0 (TID 68) in 1448 ms on localhost (executor driver) (1/1)
17:37:47.514 INFO TaskSchedulerImpl - Removed TaskSet 30.0, whose tasks have all completed, from pool
17:37:47.514 INFO DAGScheduler - ResultStage 30 (runJob at SparkHadoopWriter.scala:83) finished in 1.457 s
17:37:47.514 INFO DAGScheduler - Job 20 is finished. Cancelling potential speculative or zombie tasks for this job
17:37:47.514 INFO TaskSchedulerImpl - Killing all running tasks in stage 30: Stage finished
17:37:47.514 INFO DAGScheduler - Job 20 finished: runJob at SparkHadoopWriter.scala:83, took 1.612143 s
17:37:47.515 INFO SparkHadoopWriter - Start to commit write Job job_202510271737459220188902038722646_0082.
17:37:47.518 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=listStatus src=/user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam.parts/_temporary/0 dst=null perm=null proto=rpc
17:37:47.520 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam.parts dst=null perm=null proto=rpc
17:37:47.521 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=listStatus src=/user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam.parts/_temporary/0/task_202510271737459220188902038722646_0082_r_000000 dst=null perm=null proto=rpc
17:37:47.522 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam.parts/.part-r-00000.bai dst=null perm=null proto=rpc
17:37:47.523 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=rename src=/user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam.parts/_temporary/0/task_202510271737459220188902038722646_0082_r_000000/.part-r-00000.bai dst=/user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam.parts/.part-r-00000.bai perm=runner:supergroup:rw-r--r-- proto=rpc
17:37:47.523 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam.parts/.part-r-00000.sbi dst=null perm=null proto=rpc
17:37:47.524 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=rename src=/user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam.parts/_temporary/0/task_202510271737459220188902038722646_0082_r_000000/.part-r-00000.sbi dst=/user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam.parts/.part-r-00000.sbi perm=runner:supergroup:rw-r--r-- proto=rpc
17:37:47.525 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam.parts/part-r-00000 dst=null perm=null proto=rpc
17:37:47.531 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=rename src=/user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam.parts/_temporary/0/task_202510271737459220188902038722646_0082_r_000000/part-r-00000 dst=/user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam.parts/part-r-00000 perm=runner:supergroup:rw-r--r-- proto=rpc
17:37:47.540 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=delete src=/user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam.parts/_temporary dst=null perm=null proto=rpc
17:37:47.542 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam.parts/_SUCCESS dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:37:47.544 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam.parts/_SUCCESS is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:37:47.546 INFO audit - allowed=false ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=delete src=/user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam.parts/.spark-staging-82 dst=null perm=null proto=rpc
17:37:47.547 INFO SparkHadoopWriter - Write Job job_202510271737459220188902038722646_0082 committed. Elapsed time: 30 ms.
17:37:47.548 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam.parts/header dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:37:47.548 WARN IntelDeflaterFactory - Intel Deflater not supported, using Java.util.zip.Deflater
17:37:47.550 INFO StateChange - BLOCK* allocate blk_1073741828_1004, replicas=127.0.0.1:37927 for /user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam.parts/header
17:37:47.551 INFO DataNode - Receiving BP-617366194-10.1.0.252-1761586664431:blk_1073741828_1004 src: /127.0.0.1:56464 dest: /127.0.0.1:37927
17:37:47.553 INFO clienttrace - src: /127.0.0.1:56464, dest: /127.0.0.1:37927, bytes: 5901, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1369393814_1, offset: 0, srvID: 379dd5e5-59c3-42cd-b9d3-67ca5e44560b, blockid: BP-617366194-10.1.0.252-1761586664431:blk_1073741828_1004, duration(ns): 780682
17:37:47.553 INFO DataNode - PacketResponder: BP-617366194-10.1.0.252-1761586664431:blk_1073741828_1004, type=LAST_IN_PIPELINE terminating
17:37:47.554 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam.parts/header is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:37:47.555 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam.parts/terminator dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:37:47.557 INFO StateChange - BLOCK* allocate blk_1073741829_1005, replicas=127.0.0.1:37927 for /user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam.parts/terminator
17:37:47.558 INFO DataNode - Receiving BP-617366194-10.1.0.252-1761586664431:blk_1073741829_1005 src: /127.0.0.1:56468 dest: /127.0.0.1:37927
17:37:47.559 INFO clienttrace - src: /127.0.0.1:56468, dest: /127.0.0.1:37927, bytes: 28, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1369393814_1, offset: 0, srvID: 379dd5e5-59c3-42cd-b9d3-67ca5e44560b, blockid: BP-617366194-10.1.0.252-1761586664431:blk_1073741829_1005, duration(ns): 646583
17:37:47.559 INFO DataNode - PacketResponder: BP-617366194-10.1.0.252-1761586664431:blk_1073741829_1005, type=LAST_IN_PIPELINE terminating
17:37:47.560 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam.parts/terminator is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:37:47.561 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=listStatus src=/user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam.parts dst=null perm=null proto=rpc
17:37:47.565 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam.parts/output dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:37:47.566 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam.parts/output is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:37:47.566 INFO HadoopFileSystemWrapper - Concatenating 3 parts to hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam
17:37:47.569 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=concat src=[/user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam.parts/header, /user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam.parts/part-r-00000, /user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam.parts/terminator] dst=/user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam.parts/output perm=runner:supergroup:rw-r--r-- proto=rpc
17:37:47.570 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam dst=null perm=null proto=rpc
17:37:47.571 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=rename src=/user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam.parts/output dst=/user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam perm=runner:supergroup:rw-r--r-- proto=rpc
17:37:47.571 INFO HadoopFileSystemWrapper - Concatenating to hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam done
17:37:47.571 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam dst=null perm=null proto=rpc
17:37:47.573 INFO IndexFileMerger - Merging .sbi files in temp directory hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam.parts/ to hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam.sbi
17:37:47.573 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=listStatus src=/user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam.parts dst=null perm=null proto=rpc
17:37:47.575 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam.sbi dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:37:47.577 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam.parts/.part-r-00000.sbi dst=null perm=null proto=rpc
17:37:47.581 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam.parts/.part-r-00000.sbi dst=null perm=null proto=rpc
17:37:47.610 WARN DFSUtil - Unexpected value for data transfer bytes=216 duration=0
17:37:47.612 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=delete src=/user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam.parts/.part-r-00000.sbi dst=null perm=null proto=rpc
17:37:47.614 INFO StateChange - BLOCK* allocate blk_1073741830_1006, replicas=127.0.0.1:37927 for /user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam.sbi
17:37:47.615 INFO DataNode - Receiving BP-617366194-10.1.0.252-1761586664431:blk_1073741830_1006 src: /127.0.0.1:56482 dest: /127.0.0.1:37927
17:37:47.616 INFO clienttrace - src: /127.0.0.1:56482, dest: /127.0.0.1:37927, bytes: 212, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1369393814_1, offset: 0, srvID: 379dd5e5-59c3-42cd-b9d3-67ca5e44560b, blockid: BP-617366194-10.1.0.252-1761586664431:blk_1073741830_1006, duration(ns): 762113
17:37:47.616 INFO DataNode - PacketResponder: BP-617366194-10.1.0.252-1761586664431:blk_1073741830_1006, type=LAST_IN_PIPELINE terminating
17:37:47.618 INFO FSNamesystem - BLOCK* blk_1073741830_1006 is COMMITTED but not COMPLETE(numNodes= 0 < minimum = 1) in file /user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam.sbi
17:37:48.019 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam.sbi is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:37:48.020 INFO IndexFileMerger - Done merging .sbi files
17:37:48.021 INFO IndexFileMerger - Merging .bai files in temp directory hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam.parts/ to hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam.bai
17:37:48.022 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=listStatus src=/user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam.parts dst=null perm=null proto=rpc
17:37:48.023 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam.bai dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:37:48.025 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam.parts/.part-r-00000.bai dst=null perm=null proto=rpc
17:37:48.026 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam.parts/.part-r-00000.bai dst=null perm=null proto=rpc
17:37:48.028 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=delete src=/user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam.parts/.part-r-00000.bai dst=null perm=null proto=rpc
17:37:48.035 INFO StateChange - BLOCK* allocate blk_1073741831_1007, replicas=127.0.0.1:37927 for /user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam.bai
17:37:48.036 INFO DataNode - Receiving BP-617366194-10.1.0.252-1761586664431:blk_1073741831_1007 src: /127.0.0.1:56488 dest: /127.0.0.1:37927
17:37:48.038 INFO clienttrace - src: /127.0.0.1:56488, dest: /127.0.0.1:37927, bytes: 5472, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1369393814_1, offset: 0, srvID: 379dd5e5-59c3-42cd-b9d3-67ca5e44560b, blockid: BP-617366194-10.1.0.252-1761586664431:blk_1073741831_1007, duration(ns): 714025
17:37:48.038 INFO DataNode - PacketResponder: BP-617366194-10.1.0.252-1761586664431:blk_1073741831_1007, type=LAST_IN_PIPELINE terminating
17:37:48.039 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam.bai is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:37:48.039 INFO IndexFileMerger - Done merging .bai files
17:37:48.040 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=delete src=/user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam.parts dst=null perm=null proto=rpc
17:37:48.049 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam.bai dst=null perm=null proto=rpc
17:37:48.057 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam.sbi dst=null perm=null proto=rpc
17:37:48.058 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam.sbi dst=null perm=null proto=rpc
17:37:48.059 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam.sbi dst=null perm=null proto=rpc
17:37:48.061 WARN DFSUtil - Unexpected value for data transfer bytes=216 duration=0
17:37:48.062 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam dst=null perm=null proto=rpc
17:37:48.063 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam dst=null perm=null proto=rpc
17:37:48.063 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam dst=null perm=null proto=rpc
17:37:48.064 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam dst=null perm=null proto=rpc
17:37:48.065 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam.bai dst=null perm=null proto=rpc
17:37:48.066 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam.bai dst=null perm=null proto=rpc
17:37:48.067 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam.bai dst=null perm=null proto=rpc
17:37:48.068 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:37:48.069 WARN DFSUtil - Unexpected value for data transfer bytes=5949 duration=0
17:37:48.072 WARN DFSUtil - Unexpected value for data transfer bytes=5516 duration=0
17:37:48.073 WARN DFSUtil - Unexpected value for data transfer bytes=5516 duration=0
17:37:48.073 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:48.073 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam.sbi dst=null perm=null proto=rpc
17:37:48.074 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam.sbi dst=null perm=null proto=rpc
17:37:48.075 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam.sbi dst=null perm=null proto=rpc
17:37:48.076 WARN DFSUtil - Unexpected value for data transfer bytes=216 duration=0
17:37:48.077 INFO MemoryStore - Block broadcast_40 stored as values in memory (estimated size 320.0 B, free 1918.0 MiB)
17:37:48.078 INFO MemoryStore - Block broadcast_40_piece0 stored as bytes in memory (estimated size 233.0 B, free 1918.0 MiB)
17:37:48.078 INFO BlockManagerInfo - Added broadcast_40_piece0 in memory on localhost:37953 (size: 233.0 B, free: 1919.7 MiB)
17:37:48.079 INFO SparkContext - Created broadcast 40 from broadcast at BamSource.java:104
17:37:48.081 INFO MemoryStore - Block broadcast_41 stored as values in memory (estimated size 297.9 KiB, free 1917.7 MiB)
17:37:48.087 INFO MemoryStore - Block broadcast_41_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1917.7 MiB)
17:37:48.087 INFO BlockManagerInfo - Added broadcast_41_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.6 MiB)
17:37:48.087 INFO SparkContext - Created broadcast 41 from newAPIHadoopFile at PathSplitSource.java:96
17:37:48.102 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam dst=null perm=null proto=rpc
17:37:48.103 INFO FileInputFormat - Total input files to process : 1
17:37:48.104 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam dst=null perm=null proto=rpc
17:37:48.121 INFO SparkContext - Starting job: collect at ReadsSparkSinkUnitTest.java:182
17:37:48.122 INFO DAGScheduler - Got job 21 (collect at ReadsSparkSinkUnitTest.java:182) with 1 output partitions
17:37:48.122 INFO DAGScheduler - Final stage: ResultStage 31 (collect at ReadsSparkSinkUnitTest.java:182)
17:37:48.122 INFO DAGScheduler - Parents of final stage: List()
17:37:48.122 INFO DAGScheduler - Missing parents: List()
17:37:48.122 INFO DAGScheduler - Submitting ResultStage 31 (MapPartitionsRDD[88] at filter at ReadsSparkSource.java:96), which has no missing parents
17:37:48.129 INFO MemoryStore - Block broadcast_42 stored as values in memory (estimated size 148.2 KiB, free 1917.5 MiB)
17:37:48.130 INFO MemoryStore - Block broadcast_42_piece0 stored as bytes in memory (estimated size 54.6 KiB, free 1917.5 MiB)
17:37:48.130 INFO BlockManagerInfo - Added broadcast_42_piece0 in memory on localhost:37953 (size: 54.6 KiB, free: 1919.6 MiB)
17:37:48.130 INFO SparkContext - Created broadcast 42 from broadcast at DAGScheduler.scala:1580
17:37:48.130 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 31 (MapPartitionsRDD[88] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:37:48.131 INFO TaskSchedulerImpl - Adding task set 31.0 with 1 tasks resource profile 0
17:37:48.131 INFO TaskSetManager - Starting task 0.0 in stage 31.0 (TID 69) (localhost, executor driver, partition 0, ANY, 7852 bytes)
17:37:48.132 INFO Executor - Running task 0.0 in stage 31.0 (TID 69)
17:37:48.149 INFO NewHadoopRDD - Input split: hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam:0+238969
17:37:48.151 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam dst=null perm=null proto=rpc
17:37:48.152 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam dst=null perm=null proto=rpc
17:37:48.153 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam.bai dst=null perm=null proto=rpc
17:37:48.154 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam.bai dst=null perm=null proto=rpc
17:37:48.155 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam.bai dst=null perm=null proto=rpc
17:37:48.156 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:37:48.157 WARN DFSUtil - Unexpected value for data transfer bytes=5949 duration=0
17:37:48.160 WARN DFSUtil - Unexpected value for data transfer bytes=5516 duration=0
17:37:48.163 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:48.168 INFO Executor - Finished task 0.0 in stage 31.0 (TID 69). 651526 bytes result sent to driver
17:37:48.171 INFO TaskSetManager - Finished task 0.0 in stage 31.0 (TID 69) in 40 ms on localhost (executor driver) (1/1)
17:37:48.172 INFO TaskSchedulerImpl - Removed TaskSet 31.0, whose tasks have all completed, from pool
17:37:48.172 INFO DAGScheduler - ResultStage 31 (collect at ReadsSparkSinkUnitTest.java:182) finished in 0.050 s
17:37:48.172 INFO DAGScheduler - Job 21 is finished. Cancelling potential speculative or zombie tasks for this job
17:37:48.172 INFO TaskSchedulerImpl - Killing all running tasks in stage 31: Stage finished
17:37:48.172 INFO DAGScheduler - Job 21 finished: collect at ReadsSparkSinkUnitTest.java:182, took 0.050976 s
17:37:48.202 INFO BlockManagerInfo - Removed broadcast_38_piece0 on localhost:37953 in memory (size: 166.1 KiB, free: 1919.7 MiB)
17:37:48.203 INFO BlockManagerInfo - Removed broadcast_39_piece0 on localhost:37953 in memory (size: 67.1 KiB, free: 1919.8 MiB)
17:37:48.206 INFO BlockManagerInfo - Removed broadcast_36_piece0 on localhost:37953 in memory (size: 9.6 KiB, free: 1919.8 MiB)
17:37:48.207 INFO SparkContext - Starting job: count at ReadsSparkSinkUnitTest.java:185
17:37:48.208 INFO DAGScheduler - Got job 22 (count at ReadsSparkSinkUnitTest.java:185) with 1 output partitions
17:37:48.208 INFO DAGScheduler - Final stage: ResultStage 32 (count at ReadsSparkSinkUnitTest.java:185)
17:37:48.208 INFO DAGScheduler - Parents of final stage: List()
17:37:48.208 INFO DAGScheduler - Missing parents: List()
17:37:48.208 INFO DAGScheduler - Submitting ResultStage 32 (MapPartitionsRDD[70] at filter at ReadsSparkSource.java:96), which has no missing parents
17:37:48.210 INFO BlockManagerInfo - Removed broadcast_37_piece0 on localhost:37953 in memory (size: 9.6 KiB, free: 1919.8 MiB)
17:37:48.213 INFO BlockManagerInfo - Removed broadcast_42_piece0 on localhost:37953 in memory (size: 54.6 KiB, free: 1919.9 MiB)
17:37:48.216 INFO BlockManagerInfo - Removed broadcast_35_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.9 MiB)
17:37:48.226 INFO MemoryStore - Block broadcast_43 stored as values in memory (estimated size 426.1 KiB, free 1918.9 MiB)
17:37:48.227 INFO MemoryStore - Block broadcast_43_piece0 stored as bytes in memory (estimated size 153.6 KiB, free 1918.8 MiB)
17:37:48.227 INFO BlockManagerInfo - Added broadcast_43_piece0 in memory on localhost:37953 (size: 153.6 KiB, free: 1919.8 MiB)
17:37:48.228 INFO SparkContext - Created broadcast 43 from broadcast at DAGScheduler.scala:1580
17:37:48.228 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 32 (MapPartitionsRDD[70] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:37:48.228 INFO TaskSchedulerImpl - Adding task set 32.0 with 1 tasks resource profile 0
17:37:48.229 INFO TaskSetManager - Starting task 0.0 in stage 32.0 (TID 70) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7872 bytes)
17:37:48.229 INFO Executor - Running task 0.0 in stage 32.0 (TID 70)
17:37:48.282 INFO NewHadoopRDD - Input split: file:/home/runner/work/gatk/gatk/src/test/resources/org/broadinstitute/hellbender/tools/BQSR/HiSeq.1mb.1RG.2k_lines.bam:0+222075
17:37:48.283 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:37:48.285 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:37:48.293 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:37:48.299 INFO Executor - Finished task 0.0 in stage 32.0 (TID 70). 989 bytes result sent to driver
17:37:48.299 INFO TaskSetManager - Finished task 0.0 in stage 32.0 (TID 70) in 70 ms on localhost (executor driver) (1/1)
17:37:48.299 INFO TaskSchedulerImpl - Removed TaskSet 32.0, whose tasks have all completed, from pool
17:37:48.300 INFO DAGScheduler - ResultStage 32 (count at ReadsSparkSinkUnitTest.java:185) finished in 0.091 s
17:37:48.300 INFO DAGScheduler - Job 22 is finished. Cancelling potential speculative or zombie tasks for this job
17:37:48.300 INFO TaskSchedulerImpl - Killing all running tasks in stage 32: Stage finished
17:37:48.300 INFO DAGScheduler - Job 22 finished: count at ReadsSparkSinkUnitTest.java:185, took 0.092227 s
17:37:48.306 INFO SparkContext - Starting job: count at ReadsSparkSinkUnitTest.java:185
17:37:48.306 INFO DAGScheduler - Got job 23 (count at ReadsSparkSinkUnitTest.java:185) with 1 output partitions
17:37:48.306 INFO DAGScheduler - Final stage: ResultStage 33 (count at ReadsSparkSinkUnitTest.java:185)
17:37:48.307 INFO DAGScheduler - Parents of final stage: List()
17:37:48.307 INFO DAGScheduler - Missing parents: List()
17:37:48.307 INFO DAGScheduler - Submitting ResultStage 33 (MapPartitionsRDD[88] at filter at ReadsSparkSource.java:96), which has no missing parents
17:37:48.315 INFO MemoryStore - Block broadcast_44 stored as values in memory (estimated size 148.1 KiB, free 1918.6 MiB)
17:37:48.316 INFO MemoryStore - Block broadcast_44_piece0 stored as bytes in memory (estimated size 54.6 KiB, free 1918.6 MiB)
17:37:48.316 INFO BlockManagerInfo - Added broadcast_44_piece0 in memory on localhost:37953 (size: 54.6 KiB, free: 1919.7 MiB)
17:37:48.316 INFO SparkContext - Created broadcast 44 from broadcast at DAGScheduler.scala:1580
17:37:48.317 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 33 (MapPartitionsRDD[88] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:37:48.317 INFO TaskSchedulerImpl - Adding task set 33.0 with 1 tasks resource profile 0
17:37:48.318 INFO TaskSetManager - Starting task 0.0 in stage 33.0 (TID 71) (localhost, executor driver, partition 0, ANY, 7852 bytes)
17:37:48.318 INFO Executor - Running task 0.0 in stage 33.0 (TID 71)
17:37:48.335 INFO NewHadoopRDD - Input split: hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam:0+238969
17:37:48.336 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam dst=null perm=null proto=rpc
17:37:48.337 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam dst=null perm=null proto=rpc
17:37:48.338 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam.bai dst=null perm=null proto=rpc
17:37:48.339 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam.bai dst=null perm=null proto=rpc
17:37:48.340 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_663241b5-17b5-4aa0-abde-96895997428c.bam.bai dst=null perm=null proto=rpc
17:37:48.341 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:37:48.342 WARN DFSUtil - Unexpected value for data transfer bytes=5949 duration=0
17:37:48.344 WARN DFSUtil - Unexpected value for data transfer bytes=5516 duration=0
17:37:48.345 WARN DFSUtil - Unexpected value for data transfer bytes=5516 duration=0
17:37:48.348 WARN DFSUtil - Unexpected value for data transfer bytes=32 duration=0
17:37:48.350 INFO Executor - Finished task 0.0 in stage 33.0 (TID 71). 989 bytes result sent to driver
17:37:48.350 INFO TaskSetManager - Finished task 0.0 in stage 33.0 (TID 71) in 33 ms on localhost (executor driver) (1/1)
17:37:48.350 INFO TaskSchedulerImpl - Removed TaskSet 33.0, whose tasks have all completed, from pool
17:37:48.351 INFO DAGScheduler - ResultStage 33 (count at ReadsSparkSinkUnitTest.java:185) finished in 0.044 s
17:37:48.351 INFO DAGScheduler - Job 23 is finished. Cancelling potential speculative or zombie tasks for this job
17:37:48.351 INFO TaskSchedulerImpl - Killing all running tasks in stage 33: Stage finished
17:37:48.351 INFO DAGScheduler - Job 23 finished: count at ReadsSparkSinkUnitTest.java:185, took 0.045100 s
17:37:48.353 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:37:48.356 INFO MemoryStore - Block broadcast_45 stored as values in memory (estimated size 297.9 KiB, free 1918.3 MiB)
17:37:48.362 INFO MemoryStore - Block broadcast_45_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1918.2 MiB)
17:37:48.362 INFO BlockManagerInfo - Added broadcast_45_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.6 MiB)
17:37:48.363 INFO SparkContext - Created broadcast 45 from newAPIHadoopFile at PathSplitSource.java:96
17:37:48.386 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:37:48.389 INFO MemoryStore - Block broadcast_46 stored as values in memory (estimated size 297.9 KiB, free 1917.9 MiB)
17:37:48.395 INFO MemoryStore - Block broadcast_46_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1917.9 MiB)
17:37:48.395 INFO BlockManagerInfo - Added broadcast_46_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.6 MiB)
17:37:48.396 INFO SparkContext - Created broadcast 46 from newAPIHadoopFile at PathSplitSource.java:96
17:37:48.419 INFO FileInputFormat - Total input files to process : 1
17:37:48.423 INFO MemoryStore - Block broadcast_47 stored as values in memory (estimated size 160.7 KiB, free 1917.7 MiB)
17:37:48.425 INFO MemoryStore - Block broadcast_47_piece0 stored as bytes in memory (estimated size 9.6 KiB, free 1917.7 MiB)
17:37:48.425 INFO BlockManagerInfo - Added broadcast_47_piece0 in memory on localhost:37953 (size: 9.6 KiB, free: 1919.6 MiB)
17:37:48.426 INFO SparkContext - Created broadcast 47 from broadcast at ReadsSparkSink.java:133
17:37:48.429 INFO MemoryStore - Block broadcast_48 stored as values in memory (estimated size 163.2 KiB, free 1917.5 MiB)
17:37:48.430 INFO MemoryStore - Block broadcast_48_piece0 stored as bytes in memory (estimated size 9.6 KiB, free 1917.5 MiB)
17:37:48.430 INFO BlockManagerInfo - Added broadcast_48_piece0 in memory on localhost:37953 (size: 9.6 KiB, free: 1919.6 MiB)
17:37:48.431 INFO SparkContext - Created broadcast 48 from broadcast at BamSink.java:76
17:37:48.434 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam.parts dst=null perm=null proto=rpc
17:37:48.435 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:37:48.435 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:37:48.435 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:37:48.436 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=mkdirs src=/user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam.parts/_temporary/0 dst=null perm=runner:supergroup:rwxr-xr-x proto=rpc
17:37:48.443 INFO SparkContext - Starting job: runJob at SparkHadoopWriter.scala:83
17:37:48.443 INFO DAGScheduler - Registering RDD 102 (mapToPair at SparkUtils.java:161) as input to shuffle 8
17:37:48.444 INFO DAGScheduler - Got job 24 (runJob at SparkHadoopWriter.scala:83) with 1 output partitions
17:37:48.444 INFO DAGScheduler - Final stage: ResultStage 35 (runJob at SparkHadoopWriter.scala:83)
17:37:48.444 INFO DAGScheduler - Parents of final stage: List(ShuffleMapStage 34)
17:37:48.444 INFO DAGScheduler - Missing parents: List(ShuffleMapStage 34)
17:37:48.444 INFO DAGScheduler - Submitting ShuffleMapStage 34 (MapPartitionsRDD[102] at mapToPair at SparkUtils.java:161), which has no missing parents
17:37:48.462 INFO MemoryStore - Block broadcast_49 stored as values in memory (estimated size 520.4 KiB, free 1917.0 MiB)
17:37:48.463 INFO MemoryStore - Block broadcast_49_piece0 stored as bytes in memory (estimated size 166.1 KiB, free 1916.9 MiB)
17:37:48.464 INFO BlockManagerInfo - Added broadcast_49_piece0 in memory on localhost:37953 (size: 166.1 KiB, free: 1919.4 MiB)
17:37:48.464 INFO SparkContext - Created broadcast 49 from broadcast at DAGScheduler.scala:1580
17:37:48.464 INFO DAGScheduler - Submitting 1 missing tasks from ShuffleMapStage 34 (MapPartitionsRDD[102] at mapToPair at SparkUtils.java:161) (first 15 tasks are for partitions Vector(0))
17:37:48.464 INFO TaskSchedulerImpl - Adding task set 34.0 with 1 tasks resource profile 0
17:37:48.465 INFO TaskSetManager - Starting task 0.0 in stage 34.0 (TID 72) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7861 bytes)
17:37:48.466 INFO Executor - Running task 0.0 in stage 34.0 (TID 72)
17:37:48.512 INFO NewHadoopRDD - Input split: file:/home/runner/work/gatk/gatk/src/test/resources/org/broadinstitute/hellbender/tools/BQSR/HiSeq.1mb.1RG.2k_lines.bam:0+222075
17:37:48.514 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:37:48.515 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:37:48.528 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:37:48.542 INFO Executor - Finished task 0.0 in stage 34.0 (TID 72). 1148 bytes result sent to driver
17:37:48.542 INFO TaskSetManager - Finished task 0.0 in stage 34.0 (TID 72) in 77 ms on localhost (executor driver) (1/1)
17:37:48.542 INFO TaskSchedulerImpl - Removed TaskSet 34.0, whose tasks have all completed, from pool
17:37:48.542 INFO DAGScheduler - ShuffleMapStage 34 (mapToPair at SparkUtils.java:161) finished in 0.097 s
17:37:48.543 INFO DAGScheduler - looking for newly runnable stages
17:37:48.543 INFO DAGScheduler - running: HashSet()
17:37:48.543 INFO DAGScheduler - waiting: HashSet(ResultStage 35)
17:37:48.543 INFO DAGScheduler - failed: HashSet()
17:37:48.543 INFO DAGScheduler - Submitting ResultStage 35 (MapPartitionsRDD[107] at mapToPair at BamSink.java:91), which has no missing parents
17:37:48.550 INFO MemoryStore - Block broadcast_50 stored as values in memory (estimated size 241.5 KiB, free 1916.6 MiB)
17:37:48.551 INFO MemoryStore - Block broadcast_50_piece0 stored as bytes in memory (estimated size 67.1 KiB, free 1916.6 MiB)
17:37:48.551 INFO BlockManagerInfo - Added broadcast_50_piece0 in memory on localhost:37953 (size: 67.1 KiB, free: 1919.4 MiB)
17:37:48.551 INFO SparkContext - Created broadcast 50 from broadcast at DAGScheduler.scala:1580
17:37:48.551 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 35 (MapPartitionsRDD[107] at mapToPair at BamSink.java:91) (first 15 tasks are for partitions Vector(0))
17:37:48.551 INFO TaskSchedulerImpl - Adding task set 35.0 with 1 tasks resource profile 0
17:37:48.552 INFO TaskSetManager - Starting task 0.0 in stage 35.0 (TID 73) (localhost, executor driver, partition 0, NODE_LOCAL, 7513 bytes)
17:37:48.552 INFO Executor - Running task 0.0 in stage 35.0 (TID 73)
17:37:48.563 INFO ShuffleBlockFetcherIterator - Getting 1 (343.8 KiB) non-empty blocks including 1 (343.8 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
17:37:48.563 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
17:37:48.591 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:37:48.591 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:37:48.591 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:37:48.592 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:37:48.592 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:37:48.592 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:37:48.593 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam.parts/_temporary/0/_temporary/attempt_202510271737485526037653471384558_0107_r_000000_0/part-r-00000 dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:37:48.594 WARN IntelDeflaterFactory - Intel Deflater not supported, using Java.util.zip.Deflater
17:37:48.595 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam.parts/_temporary/0/_temporary/attempt_202510271737485526037653471384558_0107_r_000000_0/.part-r-00000.sbi dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:37:48.597 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam.parts/_temporary/0/_temporary/attempt_202510271737485526037653471384558_0107_r_000000_0/.part-r-00000.bai dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:37:48.604 INFO FsDatasetAsyncDiskService - Scheduling blk_1073741826_1002 replica FinalizedReplica, blk_1073741826_1002, FINALIZED
getNumBytes() = 212
getBytesOnDisk() = 212
getVisibleLength()= 212
getVolume() = /tmp/minicluster_storage14473397491642437130/data/data2
getBlockURI() = file:/tmp/minicluster_storage14473397491642437130/data/data2/current/BP-617366194-10.1.0.252-1761586664431/current/finalized/subdir0/subdir0/blk_1073741826 for deletion
17:37:48.605 INFO FsDatasetAsyncDiskService - Deleted BP-617366194-10.1.0.252-1761586664431 blk_1073741826_1002 URI file:/tmp/minicluster_storage14473397491642437130/data/data2/current/BP-617366194-10.1.0.252-1761586664431/current/finalized/subdir0/subdir0/blk_1073741826
17:37:48.606 INFO StateChange - BLOCK* allocate blk_1073741832_1008, replicas=127.0.0.1:37927 for /user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam.parts/_temporary/0/_temporary/attempt_202510271737485526037653471384558_0107_r_000000_0/part-r-00000
17:37:48.609 INFO DataNode - Receiving BP-617366194-10.1.0.252-1761586664431:blk_1073741832_1008 src: /127.0.0.1:56510 dest: /127.0.0.1:37927
17:37:48.612 INFO clienttrace - src: /127.0.0.1:56510, dest: /127.0.0.1:37927, bytes: 233040, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1369393814_1, offset: 0, srvID: 379dd5e5-59c3-42cd-b9d3-67ca5e44560b, blockid: BP-617366194-10.1.0.252-1761586664431:blk_1073741832_1008, duration(ns): 1243990
17:37:48.612 INFO DataNode - PacketResponder: BP-617366194-10.1.0.252-1761586664431:blk_1073741832_1008, type=LAST_IN_PIPELINE terminating
17:37:48.613 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam.parts/_temporary/0/_temporary/attempt_202510271737485526037653471384558_0107_r_000000_0/part-r-00000 is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:37:48.614 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam.parts/_temporary/0/_temporary/attempt_202510271737485526037653471384558_0107_r_000000_0/part-r-00000 dst=null perm=null proto=rpc
17:37:48.615 INFO StateChange - BLOCK* allocate blk_1073741833_1009, replicas=127.0.0.1:37927 for /user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam.parts/_temporary/0/_temporary/attempt_202510271737485526037653471384558_0107_r_000000_0/.part-r-00000.sbi
17:37:48.616 INFO DataNode - Receiving BP-617366194-10.1.0.252-1761586664431:blk_1073741833_1009 src: /127.0.0.1:56526 dest: /127.0.0.1:37927
17:37:48.617 INFO clienttrace - src: /127.0.0.1:56526, dest: /127.0.0.1:37927, bytes: 13492, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1369393814_1, offset: 0, srvID: 379dd5e5-59c3-42cd-b9d3-67ca5e44560b, blockid: BP-617366194-10.1.0.252-1761586664431:blk_1073741833_1009, duration(ns): 644252
17:37:48.617 INFO DataNode - PacketResponder: BP-617366194-10.1.0.252-1761586664431:blk_1073741833_1009, type=LAST_IN_PIPELINE terminating
17:37:48.618 INFO FSNamesystem - BLOCK* blk_1073741833_1009 is COMMITTED but not COMPLETE(numNodes= 0 < minimum = 1) in file /user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam.parts/_temporary/0/_temporary/attempt_202510271737485526037653471384558_0107_r_000000_0/.part-r-00000.sbi
17:37:49.020 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam.parts/_temporary/0/_temporary/attempt_202510271737485526037653471384558_0107_r_000000_0/.part-r-00000.sbi is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:37:49.022 INFO StateChange - BLOCK* allocate blk_1073741834_1010, replicas=127.0.0.1:37927 for /user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam.parts/_temporary/0/_temporary/attempt_202510271737485526037653471384558_0107_r_000000_0/.part-r-00000.bai
17:37:49.024 INFO DataNode - Receiving BP-617366194-10.1.0.252-1761586664431:blk_1073741834_1010 src: /127.0.0.1:56528 dest: /127.0.0.1:37927
17:37:49.025 INFO clienttrace - src: /127.0.0.1:56528, dest: /127.0.0.1:37927, bytes: 5472, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1369393814_1, offset: 0, srvID: 379dd5e5-59c3-42cd-b9d3-67ca5e44560b, blockid: BP-617366194-10.1.0.252-1761586664431:blk_1073741834_1010, duration(ns): 692901
17:37:49.025 INFO DataNode - PacketResponder: BP-617366194-10.1.0.252-1761586664431:blk_1073741834_1010, type=LAST_IN_PIPELINE terminating
17:37:49.026 INFO FSNamesystem - BLOCK* blk_1073741834_1010 is COMMITTED but not COMPLETE(numNodes= 0 < minimum = 1) in file /user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam.parts/_temporary/0/_temporary/attempt_202510271737485526037653471384558_0107_r_000000_0/.part-r-00000.bai
17:37:49.427 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam.parts/_temporary/0/_temporary/attempt_202510271737485526037653471384558_0107_r_000000_0/.part-r-00000.bai is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:37:49.428 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam.parts/_temporary/0/_temporary/attempt_202510271737485526037653471384558_0107_r_000000_0 dst=null perm=null proto=rpc
17:37:49.430 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam.parts/_temporary/0/_temporary/attempt_202510271737485526037653471384558_0107_r_000000_0 dst=null perm=null proto=rpc
17:37:49.430 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam.parts/_temporary/0/task_202510271737485526037653471384558_0107_r_000000 dst=null perm=null proto=rpc
17:37:49.431 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=rename src=/user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam.parts/_temporary/0/_temporary/attempt_202510271737485526037653471384558_0107_r_000000_0 dst=/user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam.parts/_temporary/0/task_202510271737485526037653471384558_0107_r_000000 perm=runner:supergroup:rwxr-xr-x proto=rpc
17:37:49.432 INFO FileOutputCommitter - Saved output of task 'attempt_202510271737485526037653471384558_0107_r_000000_0' to hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam.parts/_temporary/0/task_202510271737485526037653471384558_0107_r_000000
17:37:49.432 INFO SparkHadoopMapRedUtil - attempt_202510271737485526037653471384558_0107_r_000000_0: Committed. Elapsed time: 2 ms.
17:37:49.433 INFO Executor - Finished task 0.0 in stage 35.0 (TID 73). 1858 bytes result sent to driver
17:37:49.434 INFO TaskSetManager - Finished task 0.0 in stage 35.0 (TID 73) in 882 ms on localhost (executor driver) (1/1)
17:37:49.434 INFO TaskSchedulerImpl - Removed TaskSet 35.0, whose tasks have all completed, from pool
17:37:49.434 INFO DAGScheduler - ResultStage 35 (runJob at SparkHadoopWriter.scala:83) finished in 0.891 s
17:37:49.435 INFO DAGScheduler - Job 24 is finished. Cancelling potential speculative or zombie tasks for this job
17:37:49.435 INFO TaskSchedulerImpl - Killing all running tasks in stage 35: Stage finished
17:37:49.435 INFO DAGScheduler - Job 24 finished: runJob at SparkHadoopWriter.scala:83, took 0.991976 s
17:37:49.436 INFO SparkHadoopWriter - Start to commit write Job job_202510271737485526037653471384558_0107.
17:37:49.436 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=listStatus src=/user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam.parts/_temporary/0 dst=null perm=null proto=rpc
17:37:49.437 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam.parts dst=null perm=null proto=rpc
17:37:49.438 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=listStatus src=/user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam.parts/_temporary/0/task_202510271737485526037653471384558_0107_r_000000 dst=null perm=null proto=rpc
17:37:49.439 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam.parts/.part-r-00000.bai dst=null perm=null proto=rpc
17:37:49.440 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=rename src=/user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam.parts/_temporary/0/task_202510271737485526037653471384558_0107_r_000000/.part-r-00000.bai dst=/user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam.parts/.part-r-00000.bai perm=runner:supergroup:rw-r--r-- proto=rpc
17:37:49.440 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam.parts/.part-r-00000.sbi dst=null perm=null proto=rpc
17:37:49.441 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=rename src=/user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam.parts/_temporary/0/task_202510271737485526037653471384558_0107_r_000000/.part-r-00000.sbi dst=/user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam.parts/.part-r-00000.sbi perm=runner:supergroup:rw-r--r-- proto=rpc
17:37:49.442 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam.parts/part-r-00000 dst=null perm=null proto=rpc
17:37:49.443 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=rename src=/user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam.parts/_temporary/0/task_202510271737485526037653471384558_0107_r_000000/part-r-00000 dst=/user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam.parts/part-r-00000 perm=runner:supergroup:rw-r--r-- proto=rpc
17:37:49.444 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=delete src=/user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam.parts/_temporary dst=null perm=null proto=rpc
17:37:49.445 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam.parts/_SUCCESS dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:37:49.446 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam.parts/_SUCCESS is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:37:49.447 INFO audit - allowed=false ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=delete src=/user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam.parts/.spark-staging-107 dst=null perm=null proto=rpc
17:37:49.447 INFO SparkHadoopWriter - Write Job job_202510271737485526037653471384558_0107 committed. Elapsed time: 11 ms.
17:37:49.448 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam.parts/header dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:37:49.448 WARN IntelDeflaterFactory - Intel Deflater not supported, using Java.util.zip.Deflater
17:37:49.450 INFO StateChange - BLOCK* allocate blk_1073741835_1011, replicas=127.0.0.1:37927 for /user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam.parts/header
17:37:49.451 INFO DataNode - Receiving BP-617366194-10.1.0.252-1761586664431:blk_1073741835_1011 src: /127.0.0.1:56544 dest: /127.0.0.1:37927
17:37:49.452 INFO clienttrace - src: /127.0.0.1:56544, dest: /127.0.0.1:37927, bytes: 5901, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1369393814_1, offset: 0, srvID: 379dd5e5-59c3-42cd-b9d3-67ca5e44560b, blockid: BP-617366194-10.1.0.252-1761586664431:blk_1073741835_1011, duration(ns): 603506
17:37:49.452 INFO DataNode - PacketResponder: BP-617366194-10.1.0.252-1761586664431:blk_1073741835_1011, type=LAST_IN_PIPELINE terminating
17:37:49.453 INFO FSNamesystem - BLOCK* blk_1073741835_1011 is COMMITTED but not COMPLETE(numNodes= 0 < minimum = 1) in file /user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam.parts/header
17:37:49.855 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam.parts/header is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:37:49.856 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam.parts/terminator dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:37:49.857 INFO StateChange - BLOCK* allocate blk_1073741836_1012, replicas=127.0.0.1:37927 for /user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam.parts/terminator
17:37:49.859 INFO DataNode - Receiving BP-617366194-10.1.0.252-1761586664431:blk_1073741836_1012 src: /127.0.0.1:56556 dest: /127.0.0.1:37927
17:37:49.860 INFO clienttrace - src: /127.0.0.1:56556, dest: /127.0.0.1:37927, bytes: 28, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1369393814_1, offset: 0, srvID: 379dd5e5-59c3-42cd-b9d3-67ca5e44560b, blockid: BP-617366194-10.1.0.252-1761586664431:blk_1073741836_1012, duration(ns): 650068
17:37:49.860 INFO DataNode - PacketResponder: BP-617366194-10.1.0.252-1761586664431:blk_1073741836_1012, type=LAST_IN_PIPELINE terminating
17:37:49.861 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam.parts/terminator is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:37:49.862 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=listStatus src=/user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam.parts dst=null perm=null proto=rpc
17:37:49.864 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam.parts/output dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:37:49.865 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam.parts/output is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:37:49.865 INFO HadoopFileSystemWrapper - Concatenating 3 parts to hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam
17:37:49.866 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=concat src=[/user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam.parts/header, /user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam.parts/part-r-00000, /user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam.parts/terminator] dst=/user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam.parts/output perm=runner:supergroup:rw-r--r-- proto=rpc
17:37:49.866 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam dst=null perm=null proto=rpc
17:37:49.867 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=rename src=/user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam.parts/output dst=/user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam perm=runner:supergroup:rw-r--r-- proto=rpc
17:37:49.867 INFO HadoopFileSystemWrapper - Concatenating to hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam done
17:37:49.868 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam dst=null perm=null proto=rpc
17:37:49.868 INFO IndexFileMerger - Merging .sbi files in temp directory hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam.parts/ to hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam.sbi
17:37:49.869 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=listStatus src=/user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam.parts dst=null perm=null proto=rpc
17:37:49.870 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam.sbi dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:37:49.871 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam.parts/.part-r-00000.sbi dst=null perm=null proto=rpc
17:37:49.872 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam.parts/.part-r-00000.sbi dst=null perm=null proto=rpc
17:37:49.873 WARN DFSUtil - Unexpected value for data transfer bytes=13600 duration=0
17:37:49.874 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=delete src=/user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam.parts/.part-r-00000.sbi dst=null perm=null proto=rpc
17:37:49.876 INFO StateChange - BLOCK* allocate blk_1073741837_1013, replicas=127.0.0.1:37927 for /user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam.sbi
17:37:49.876 INFO DataNode - Receiving BP-617366194-10.1.0.252-1761586664431:blk_1073741837_1013 src: /127.0.0.1:56572 dest: /127.0.0.1:37927
17:37:49.878 INFO clienttrace - src: /127.0.0.1:56572, dest: /127.0.0.1:37927, bytes: 13492, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1369393814_1, offset: 0, srvID: 379dd5e5-59c3-42cd-b9d3-67ca5e44560b, blockid: BP-617366194-10.1.0.252-1761586664431:blk_1073741837_1013, duration(ns): 745079
17:37:49.878 INFO DataNode - PacketResponder: BP-617366194-10.1.0.252-1761586664431:blk_1073741837_1013, type=LAST_IN_PIPELINE terminating
17:37:49.879 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam.sbi is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:37:49.879 INFO IndexFileMerger - Done merging .sbi files
17:37:49.880 INFO IndexFileMerger - Merging .bai files in temp directory hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam.parts/ to hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam.bai
17:37:49.880 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=listStatus src=/user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam.parts dst=null perm=null proto=rpc
17:37:49.881 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam.bai dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:37:49.882 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam.parts/.part-r-00000.bai dst=null perm=null proto=rpc
17:37:49.883 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam.parts/.part-r-00000.bai dst=null perm=null proto=rpc
17:37:49.884 WARN DFSUtil - Unexpected value for data transfer bytes=5516 duration=0
17:37:49.885 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=delete src=/user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam.parts/.part-r-00000.bai dst=null perm=null proto=rpc
17:37:49.889 INFO StateChange - BLOCK* allocate blk_1073741838_1014, replicas=127.0.0.1:37927 for /user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam.bai
17:37:49.890 INFO DataNode - Receiving BP-617366194-10.1.0.252-1761586664431:blk_1073741838_1014 src: /127.0.0.1:56584 dest: /127.0.0.1:37927
17:37:49.891 INFO clienttrace - src: /127.0.0.1:56584, dest: /127.0.0.1:37927, bytes: 5472, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1369393814_1, offset: 0, srvID: 379dd5e5-59c3-42cd-b9d3-67ca5e44560b, blockid: BP-617366194-10.1.0.252-1761586664431:blk_1073741838_1014, duration(ns): 653596
17:37:49.891 INFO DataNode - PacketResponder: BP-617366194-10.1.0.252-1761586664431:blk_1073741838_1014, type=LAST_IN_PIPELINE terminating
17:37:49.892 INFO FSNamesystem - BLOCK* blk_1073741838_1014 is COMMITTED but not COMPLETE(numNodes= 0 < minimum = 1) in file /user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam.bai
17:37:50.293 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam.bai is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:37:50.294 INFO IndexFileMerger - Done merging .bai files
17:37:50.295 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=delete src=/user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam.parts dst=null perm=null proto=rpc
17:37:50.304 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam.bai dst=null perm=null proto=rpc
17:37:50.312 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam.sbi dst=null perm=null proto=rpc
17:37:50.313 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam.sbi dst=null perm=null proto=rpc
17:37:50.313 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam.sbi dst=null perm=null proto=rpc
17:37:50.315 WARN DFSUtil - Unexpected value for data transfer bytes=13600 duration=0
17:37:50.316 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam dst=null perm=null proto=rpc
17:37:50.317 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam dst=null perm=null proto=rpc
17:37:50.317 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam dst=null perm=null proto=rpc
17:37:50.318 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam dst=null perm=null proto=rpc
17:37:50.319 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam.bai dst=null perm=null proto=rpc
17:37:50.320 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam.bai dst=null perm=null proto=rpc
17:37:50.321 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam.bai dst=null perm=null proto=rpc
17:37:50.322 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:37:50.323 WARN DFSUtil - Unexpected value for data transfer bytes=5949 duration=0
17:37:50.325 WARN DFSUtil - Unexpected value for data transfer bytes=5516 duration=0
17:37:50.326 WARN DFSUtil - Unexpected value for data transfer bytes=5516 duration=0
17:37:50.326 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:50.327 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam.sbi dst=null perm=null proto=rpc
17:37:50.328 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam.sbi dst=null perm=null proto=rpc
17:37:50.328 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam.sbi dst=null perm=null proto=rpc
17:37:50.329 WARN DFSUtil - Unexpected value for data transfer bytes=13600 duration=0
17:37:50.330 INFO MemoryStore - Block broadcast_51 stored as values in memory (estimated size 13.3 KiB, free 1916.6 MiB)
17:37:50.331 INFO MemoryStore - Block broadcast_51_piece0 stored as bytes in memory (estimated size 8.3 KiB, free 1916.5 MiB)
17:37:50.331 INFO BlockManagerInfo - Added broadcast_51_piece0 in memory on localhost:37953 (size: 8.3 KiB, free: 1919.3 MiB)
17:37:50.331 INFO SparkContext - Created broadcast 51 from broadcast at BamSource.java:104
17:37:50.333 INFO MemoryStore - Block broadcast_52 stored as values in memory (estimated size 297.9 KiB, free 1916.3 MiB)
17:37:50.339 INFO MemoryStore - Block broadcast_52_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1916.2 MiB)
17:37:50.339 INFO BlockManagerInfo - Added broadcast_52_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.3 MiB)
17:37:50.339 INFO SparkContext - Created broadcast 52 from newAPIHadoopFile at PathSplitSource.java:96
17:37:50.350 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam dst=null perm=null proto=rpc
17:37:50.350 INFO FileInputFormat - Total input files to process : 1
17:37:50.351 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam dst=null perm=null proto=rpc
17:37:50.366 INFO SparkContext - Starting job: collect at ReadsSparkSinkUnitTest.java:182
17:37:50.366 INFO DAGScheduler - Got job 25 (collect at ReadsSparkSinkUnitTest.java:182) with 1 output partitions
17:37:50.366 INFO DAGScheduler - Final stage: ResultStage 36 (collect at ReadsSparkSinkUnitTest.java:182)
17:37:50.366 INFO DAGScheduler - Parents of final stage: List()
17:37:50.367 INFO DAGScheduler - Missing parents: List()
17:37:50.367 INFO DAGScheduler - Submitting ResultStage 36 (MapPartitionsRDD[113] at filter at ReadsSparkSource.java:96), which has no missing parents
17:37:50.373 INFO MemoryStore - Block broadcast_53 stored as values in memory (estimated size 148.2 KiB, free 1916.1 MiB)
17:37:50.374 INFO MemoryStore - Block broadcast_53_piece0 stored as bytes in memory (estimated size 54.6 KiB, free 1916.0 MiB)
17:37:50.374 INFO BlockManagerInfo - Added broadcast_53_piece0 in memory on localhost:37953 (size: 54.6 KiB, free: 1919.2 MiB)
17:37:50.374 INFO SparkContext - Created broadcast 53 from broadcast at DAGScheduler.scala:1580
17:37:50.374 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 36 (MapPartitionsRDD[113] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:37:50.375 INFO TaskSchedulerImpl - Adding task set 36.0 with 1 tasks resource profile 0
17:37:50.375 INFO TaskSetManager - Starting task 0.0 in stage 36.0 (TID 74) (localhost, executor driver, partition 0, ANY, 7852 bytes)
17:37:50.376 INFO Executor - Running task 0.0 in stage 36.0 (TID 74)
17:37:50.392 INFO NewHadoopRDD - Input split: hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam:0+238969
17:37:50.393 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam dst=null perm=null proto=rpc
17:37:50.394 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam dst=null perm=null proto=rpc
17:37:50.395 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam.bai dst=null perm=null proto=rpc
17:37:50.396 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam.bai dst=null perm=null proto=rpc
17:37:50.397 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam.bai dst=null perm=null proto=rpc
17:37:50.398 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:37:50.409 INFO BlockManagerInfo - Removed broadcast_34_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.3 MiB)
17:37:50.409 WARN DFSUtil - Unexpected value for data transfer bytes=5516 duration=0
17:37:50.409 WARN DFSUtil - Unexpected value for data transfer bytes=5516 duration=0
17:37:50.411 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:50.413 INFO BlockManagerInfo - Removed broadcast_43_piece0 on localhost:37953 in memory (size: 153.6 KiB, free: 1919.4 MiB)
17:37:50.414 INFO BlockManagerInfo - Removed broadcast_41_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.5 MiB)
17:37:50.414 WARN DFSUtil - Unexpected value for data transfer bytes=32 duration=0
17:37:50.416 INFO BlockManagerInfo - Removed broadcast_47_piece0 on localhost:37953 in memory (size: 9.6 KiB, free: 1919.5 MiB)
17:37:50.416 INFO Executor - Finished task 0.0 in stage 36.0 (TID 74). 651569 bytes result sent to driver
17:37:50.419 INFO TaskSetManager - Finished task 0.0 in stage 36.0 (TID 74) in 44 ms on localhost (executor driver) (1/1)
17:37:50.419 INFO DAGScheduler - ResultStage 36 (collect at ReadsSparkSinkUnitTest.java:182) finished in 0.052 s
17:37:50.419 INFO DAGScheduler - Job 25 is finished. Cancelling potential speculative or zombie tasks for this job
17:37:50.420 INFO BlockManagerInfo - Removed broadcast_50_piece0 on localhost:37953 in memory (size: 67.1 KiB, free: 1919.6 MiB)
17:37:50.420 INFO TaskSchedulerImpl - Removed TaskSet 36.0, whose tasks have all completed, from pool
17:37:50.420 INFO TaskSchedulerImpl - Killing all running tasks in stage 36: Stage finished
17:37:50.420 INFO DAGScheduler - Job 25 finished: collect at ReadsSparkSinkUnitTest.java:182, took 0.054433 s
17:37:50.421 INFO BlockManagerInfo - Removed broadcast_44_piece0 on localhost:37953 in memory (size: 54.6 KiB, free: 1919.6 MiB)
17:37:50.422 INFO BlockManagerInfo - Removed broadcast_49_piece0 on localhost:37953 in memory (size: 166.1 KiB, free: 1919.8 MiB)
17:37:50.425 INFO BlockManagerInfo - Removed broadcast_46_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.8 MiB)
17:37:50.427 INFO BlockManagerInfo - Removed broadcast_48_piece0 on localhost:37953 in memory (size: 9.6 KiB, free: 1919.8 MiB)
17:37:50.428 INFO BlockManagerInfo - Removed broadcast_40_piece0 on localhost:37953 in memory (size: 233.0 B, free: 1919.8 MiB)
17:37:50.437 INFO SparkContext - Starting job: count at ReadsSparkSinkUnitTest.java:185
17:37:50.438 INFO DAGScheduler - Got job 26 (count at ReadsSparkSinkUnitTest.java:185) with 1 output partitions
17:37:50.438 INFO DAGScheduler - Final stage: ResultStage 37 (count at ReadsSparkSinkUnitTest.java:185)
17:37:50.438 INFO DAGScheduler - Parents of final stage: List()
17:37:50.438 INFO DAGScheduler - Missing parents: List()
17:37:50.439 INFO DAGScheduler - Submitting ResultStage 37 (MapPartitionsRDD[95] at filter at ReadsSparkSource.java:96), which has no missing parents
17:37:50.456 INFO MemoryStore - Block broadcast_54 stored as values in memory (estimated size 426.1 KiB, free 1918.7 MiB)
17:37:50.458 INFO MemoryStore - Block broadcast_54_piece0 stored as bytes in memory (estimated size 153.6 KiB, free 1918.5 MiB)
17:37:50.458 INFO BlockManagerInfo - Added broadcast_54_piece0 in memory on localhost:37953 (size: 153.6 KiB, free: 1919.7 MiB)
17:37:50.458 INFO SparkContext - Created broadcast 54 from broadcast at DAGScheduler.scala:1580
17:37:50.458 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 37 (MapPartitionsRDD[95] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:37:50.458 INFO TaskSchedulerImpl - Adding task set 37.0 with 1 tasks resource profile 0
17:37:50.459 INFO TaskSetManager - Starting task 0.0 in stage 37.0 (TID 75) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7872 bytes)
17:37:50.460 INFO Executor - Running task 0.0 in stage 37.0 (TID 75)
17:37:50.503 INFO NewHadoopRDD - Input split: file:/home/runner/work/gatk/gatk/src/test/resources/org/broadinstitute/hellbender/tools/BQSR/HiSeq.1mb.1RG.2k_lines.bam:0+222075
17:37:50.504 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:37:50.505 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:37:50.514 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:37:50.519 INFO Executor - Finished task 0.0 in stage 37.0 (TID 75). 989 bytes result sent to driver
17:37:50.519 INFO TaskSetManager - Finished task 0.0 in stage 37.0 (TID 75) in 60 ms on localhost (executor driver) (1/1)
17:37:50.519 INFO TaskSchedulerImpl - Removed TaskSet 37.0, whose tasks have all completed, from pool
17:37:50.519 INFO DAGScheduler - ResultStage 37 (count at ReadsSparkSinkUnitTest.java:185) finished in 0.080 s
17:37:50.519 INFO DAGScheduler - Job 26 is finished. Cancelling potential speculative or zombie tasks for this job
17:37:50.520 INFO TaskSchedulerImpl - Killing all running tasks in stage 37: Stage finished
17:37:50.520 INFO DAGScheduler - Job 26 finished: count at ReadsSparkSinkUnitTest.java:185, took 0.082288 s
17:37:50.523 INFO SparkContext - Starting job: count at ReadsSparkSinkUnitTest.java:185
17:37:50.523 INFO DAGScheduler - Got job 27 (count at ReadsSparkSinkUnitTest.java:185) with 1 output partitions
17:37:50.524 INFO DAGScheduler - Final stage: ResultStage 38 (count at ReadsSparkSinkUnitTest.java:185)
17:37:50.524 INFO DAGScheduler - Parents of final stage: List()
17:37:50.524 INFO DAGScheduler - Missing parents: List()
17:37:50.524 INFO DAGScheduler - Submitting ResultStage 38 (MapPartitionsRDD[113] at filter at ReadsSparkSource.java:96), which has no missing parents
17:37:50.530 INFO MemoryStore - Block broadcast_55 stored as values in memory (estimated size 148.1 KiB, free 1918.4 MiB)
17:37:50.530 INFO MemoryStore - Block broadcast_55_piece0 stored as bytes in memory (estimated size 54.6 KiB, free 1918.3 MiB)
17:37:50.531 INFO BlockManagerInfo - Added broadcast_55_piece0 in memory on localhost:37953 (size: 54.6 KiB, free: 1919.6 MiB)
17:37:50.531 INFO SparkContext - Created broadcast 55 from broadcast at DAGScheduler.scala:1580
17:37:50.531 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 38 (MapPartitionsRDD[113] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:37:50.531 INFO TaskSchedulerImpl - Adding task set 38.0 with 1 tasks resource profile 0
17:37:50.532 INFO TaskSetManager - Starting task 0.0 in stage 38.0 (TID 76) (localhost, executor driver, partition 0, ANY, 7852 bytes)
17:37:50.532 INFO Executor - Running task 0.0 in stage 38.0 (TID 76)
17:37:50.548 INFO NewHadoopRDD - Input split: hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam:0+238969
17:37:50.549 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam dst=null perm=null proto=rpc
17:37:50.550 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam dst=null perm=null proto=rpc
17:37:50.551 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam.bai dst=null perm=null proto=rpc
17:37:50.552 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam.bai dst=null perm=null proto=rpc
17:37:50.552 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_2a02fd9f-64cf-49d1-a608-3c5e6e53e2ea.bam.bai dst=null perm=null proto=rpc
17:37:50.553 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:37:50.554 WARN DFSUtil - Unexpected value for data transfer bytes=5949 duration=0
17:37:50.556 WARN DFSUtil - Unexpected value for data transfer bytes=5516 duration=0
17:37:50.557 WARN DFSUtil - Unexpected value for data transfer bytes=5516 duration=0
17:37:50.559 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:50.562 INFO Executor - Finished task 0.0 in stage 38.0 (TID 76). 989 bytes result sent to driver
17:37:50.562 INFO TaskSetManager - Finished task 0.0 in stage 38.0 (TID 76) in 30 ms on localhost (executor driver) (1/1)
17:37:50.562 INFO TaskSchedulerImpl - Removed TaskSet 38.0, whose tasks have all completed, from pool
17:37:50.563 INFO DAGScheduler - ResultStage 38 (count at ReadsSparkSinkUnitTest.java:185) finished in 0.038 s
17:37:50.563 INFO DAGScheduler - Job 27 is finished. Cancelling potential speculative or zombie tasks for this job
17:37:50.563 INFO TaskSchedulerImpl - Killing all running tasks in stage 38: Stage finished
17:37:50.563 INFO DAGScheduler - Job 27 finished: count at ReadsSparkSinkUnitTest.java:185, took 0.039619 s
17:37:50.564 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:37:50.567 INFO MemoryStore - Block broadcast_56 stored as values in memory (estimated size 297.9 KiB, free 1918.0 MiB)
17:37:50.573 INFO MemoryStore - Block broadcast_56_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1918.0 MiB)
17:37:50.573 INFO BlockManagerInfo - Added broadcast_56_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.6 MiB)
17:37:50.573 INFO SparkContext - Created broadcast 56 from newAPIHadoopFile at PathSplitSource.java:96
17:37:50.594 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:37:50.597 INFO MemoryStore - Block broadcast_57 stored as values in memory (estimated size 297.9 KiB, free 1917.7 MiB)
17:37:50.603 INFO MemoryStore - Block broadcast_57_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1917.7 MiB)
17:37:50.603 INFO BlockManagerInfo - Added broadcast_57_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.5 MiB)
17:37:50.603 INFO SparkContext - Created broadcast 57 from newAPIHadoopFile at PathSplitSource.java:96
17:37:50.624 INFO FileInputFormat - Total input files to process : 1
17:37:50.626 INFO MemoryStore - Block broadcast_58 stored as values in memory (estimated size 160.7 KiB, free 1917.5 MiB)
17:37:50.628 INFO MemoryStore - Block broadcast_58_piece0 stored as bytes in memory (estimated size 9.6 KiB, free 1917.5 MiB)
17:37:50.628 INFO BlockManagerInfo - Added broadcast_58_piece0 in memory on localhost:37953 (size: 9.6 KiB, free: 1919.5 MiB)
17:37:50.628 INFO SparkContext - Created broadcast 58 from broadcast at ReadsSparkSink.java:133
17:37:50.629 WARN HtsjdkReadsRddStorage - Unrecognized write option: DISABLE
17:37:50.630 INFO MemoryStore - Block broadcast_59 stored as values in memory (estimated size 163.2 KiB, free 1917.3 MiB)
17:37:50.632 INFO MemoryStore - Block broadcast_59_piece0 stored as bytes in memory (estimated size 9.6 KiB, free 1917.3 MiB)
17:37:50.632 INFO BlockManagerInfo - Added broadcast_59_piece0 in memory on localhost:37953 (size: 9.6 KiB, free: 1919.5 MiB)
17:37:50.632 INFO SparkContext - Created broadcast 59 from broadcast at BamSink.java:76
17:37:50.635 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_bca83413-d6b6-4abe-a6f6-e11360f734b1.bam.parts dst=null perm=null proto=rpc
17:37:50.635 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:37:50.635 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:37:50.635 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:37:50.636 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=mkdirs src=/user/runner/ReadsSparkSinkUnitTest1_bca83413-d6b6-4abe-a6f6-e11360f734b1.bam.parts/_temporary/0 dst=null perm=runner:supergroup:rwxr-xr-x proto=rpc
17:37:50.642 INFO SparkContext - Starting job: runJob at SparkHadoopWriter.scala:83
17:37:50.643 INFO DAGScheduler - Registering RDD 127 (mapToPair at SparkUtils.java:161) as input to shuffle 9
17:37:50.643 INFO DAGScheduler - Got job 28 (runJob at SparkHadoopWriter.scala:83) with 1 output partitions
17:37:50.643 INFO DAGScheduler - Final stage: ResultStage 40 (runJob at SparkHadoopWriter.scala:83)
17:37:50.643 INFO DAGScheduler - Parents of final stage: List(ShuffleMapStage 39)
17:37:50.644 INFO DAGScheduler - Missing parents: List(ShuffleMapStage 39)
17:37:50.644 INFO DAGScheduler - Submitting ShuffleMapStage 39 (MapPartitionsRDD[127] at mapToPair at SparkUtils.java:161), which has no missing parents
17:37:50.661 INFO MemoryStore - Block broadcast_60 stored as values in memory (estimated size 520.4 KiB, free 1916.8 MiB)
17:37:50.662 INFO MemoryStore - Block broadcast_60_piece0 stored as bytes in memory (estimated size 166.1 KiB, free 1916.7 MiB)
17:37:50.662 INFO BlockManagerInfo - Added broadcast_60_piece0 in memory on localhost:37953 (size: 166.1 KiB, free: 1919.4 MiB)
17:37:50.662 INFO SparkContext - Created broadcast 60 from broadcast at DAGScheduler.scala:1580
17:37:50.663 INFO DAGScheduler - Submitting 1 missing tasks from ShuffleMapStage 39 (MapPartitionsRDD[127] at mapToPair at SparkUtils.java:161) (first 15 tasks are for partitions Vector(0))
17:37:50.663 INFO TaskSchedulerImpl - Adding task set 39.0 with 1 tasks resource profile 0
17:37:50.663 INFO TaskSetManager - Starting task 0.0 in stage 39.0 (TID 77) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7861 bytes)
17:37:50.664 INFO Executor - Running task 0.0 in stage 39.0 (TID 77)
17:37:50.702 INFO NewHadoopRDD - Input split: file:/home/runner/work/gatk/gatk/src/test/resources/org/broadinstitute/hellbender/tools/BQSR/HiSeq.1mb.1RG.2k_lines.bam:0+222075
17:37:50.703 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:37:50.704 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:37:50.712 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:37:50.725 INFO Executor - Finished task 0.0 in stage 39.0 (TID 77). 1148 bytes result sent to driver
17:37:50.725 INFO TaskSetManager - Finished task 0.0 in stage 39.0 (TID 77) in 62 ms on localhost (executor driver) (1/1)
17:37:50.726 INFO TaskSchedulerImpl - Removed TaskSet 39.0, whose tasks have all completed, from pool
17:37:50.726 INFO DAGScheduler - ShuffleMapStage 39 (mapToPair at SparkUtils.java:161) finished in 0.082 s
17:37:50.726 INFO DAGScheduler - looking for newly runnable stages
17:37:50.726 INFO DAGScheduler - running: HashSet()
17:37:50.726 INFO DAGScheduler - waiting: HashSet(ResultStage 40)
17:37:50.726 INFO DAGScheduler - failed: HashSet()
17:37:50.726 INFO DAGScheduler - Submitting ResultStage 40 (MapPartitionsRDD[132] at mapToPair at BamSink.java:91), which has no missing parents
17:37:50.733 INFO MemoryStore - Block broadcast_61 stored as values in memory (estimated size 241.5 KiB, free 1916.4 MiB)
17:37:50.734 INFO MemoryStore - Block broadcast_61_piece0 stored as bytes in memory (estimated size 67.1 KiB, free 1916.4 MiB)
17:37:50.734 INFO BlockManagerInfo - Added broadcast_61_piece0 in memory on localhost:37953 (size: 67.1 KiB, free: 1919.3 MiB)
17:37:50.734 INFO SparkContext - Created broadcast 61 from broadcast at DAGScheduler.scala:1580
17:37:50.735 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 40 (MapPartitionsRDD[132] at mapToPair at BamSink.java:91) (first 15 tasks are for partitions Vector(0))
17:37:50.735 INFO TaskSchedulerImpl - Adding task set 40.0 with 1 tasks resource profile 0
17:37:50.735 INFO TaskSetManager - Starting task 0.0 in stage 40.0 (TID 78) (localhost, executor driver, partition 0, NODE_LOCAL, 7513 bytes)
17:37:50.736 INFO Executor - Running task 0.0 in stage 40.0 (TID 78)
17:37:50.742 INFO ShuffleBlockFetcherIterator - Getting 1 (343.8 KiB) non-empty blocks including 1 (343.8 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
17:37:50.742 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
17:37:50.760 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:37:50.760 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:37:50.760 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:37:50.760 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:37:50.760 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:37:50.760 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:37:50.762 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest1_bca83413-d6b6-4abe-a6f6-e11360f734b1.bam.parts/_temporary/0/_temporary/attempt_202510271737505759150451256143092_0132_r_000000_0/part-r-00000 dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:37:50.762 WARN IntelDeflaterFactory - Intel Deflater not supported, using Java.util.zip.Deflater
17:37:50.763 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest1_bca83413-d6b6-4abe-a6f6-e11360f734b1.bam.parts/_temporary/0/_temporary/attempt_202510271737505759150451256143092_0132_r_000000_0/.part-r-00000.bai dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:37:50.768 INFO StateChange - BLOCK* allocate blk_1073741839_1015, replicas=127.0.0.1:37927 for /user/runner/ReadsSparkSinkUnitTest1_bca83413-d6b6-4abe-a6f6-e11360f734b1.bam.parts/_temporary/0/_temporary/attempt_202510271737505759150451256143092_0132_r_000000_0/part-r-00000
17:37:50.770 INFO DataNode - Receiving BP-617366194-10.1.0.252-1761586664431:blk_1073741839_1015 src: /127.0.0.1:56588 dest: /127.0.0.1:37927
17:37:50.775 INFO clienttrace - src: /127.0.0.1:56588, dest: /127.0.0.1:37927, bytes: 233040, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1369393814_1, offset: 0, srvID: 379dd5e5-59c3-42cd-b9d3-67ca5e44560b, blockid: BP-617366194-10.1.0.252-1761586664431:blk_1073741839_1015, duration(ns): 4577340
17:37:50.775 INFO DataNode - PacketResponder: BP-617366194-10.1.0.252-1761586664431:blk_1073741839_1015, type=LAST_IN_PIPELINE terminating
17:37:50.776 INFO FSNamesystem - BLOCK* blk_1073741839_1015 is COMMITTED but not COMPLETE(numNodes= 0 < minimum = 1) in file /user/runner/ReadsSparkSinkUnitTest1_bca83413-d6b6-4abe-a6f6-e11360f734b1.bam.parts/_temporary/0/_temporary/attempt_202510271737505759150451256143092_0132_r_000000_0/part-r-00000
17:37:51.179 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest1_bca83413-d6b6-4abe-a6f6-e11360f734b1.bam.parts/_temporary/0/_temporary/attempt_202510271737505759150451256143092_0132_r_000000_0/part-r-00000 is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:37:51.180 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_bca83413-d6b6-4abe-a6f6-e11360f734b1.bam.parts/_temporary/0/_temporary/attempt_202510271737505759150451256143092_0132_r_000000_0/part-r-00000 dst=null perm=null proto=rpc
17:37:51.182 INFO StateChange - BLOCK* allocate blk_1073741840_1016, replicas=127.0.0.1:37927 for /user/runner/ReadsSparkSinkUnitTest1_bca83413-d6b6-4abe-a6f6-e11360f734b1.bam.parts/_temporary/0/_temporary/attempt_202510271737505759150451256143092_0132_r_000000_0/.part-r-00000.bai
17:37:51.184 INFO DataNode - Receiving BP-617366194-10.1.0.252-1761586664431:blk_1073741840_1016 src: /127.0.0.1:56600 dest: /127.0.0.1:37927
17:37:51.185 INFO clienttrace - src: /127.0.0.1:56600, dest: /127.0.0.1:37927, bytes: 5472, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1369393814_1, offset: 0, srvID: 379dd5e5-59c3-42cd-b9d3-67ca5e44560b, blockid: BP-617366194-10.1.0.252-1761586664431:blk_1073741840_1016, duration(ns): 717350
17:37:51.185 INFO DataNode - PacketResponder: BP-617366194-10.1.0.252-1761586664431:blk_1073741840_1016, type=LAST_IN_PIPELINE terminating
17:37:51.186 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest1_bca83413-d6b6-4abe-a6f6-e11360f734b1.bam.parts/_temporary/0/_temporary/attempt_202510271737505759150451256143092_0132_r_000000_0/.part-r-00000.bai is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:37:51.187 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_bca83413-d6b6-4abe-a6f6-e11360f734b1.bam.parts/_temporary/0/_temporary/attempt_202510271737505759150451256143092_0132_r_000000_0 dst=null perm=null proto=rpc
17:37:51.188 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_bca83413-d6b6-4abe-a6f6-e11360f734b1.bam.parts/_temporary/0/_temporary/attempt_202510271737505759150451256143092_0132_r_000000_0 dst=null perm=null proto=rpc
17:37:51.189 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_bca83413-d6b6-4abe-a6f6-e11360f734b1.bam.parts/_temporary/0/task_202510271737505759150451256143092_0132_r_000000 dst=null perm=null proto=rpc
17:37:51.189 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=rename src=/user/runner/ReadsSparkSinkUnitTest1_bca83413-d6b6-4abe-a6f6-e11360f734b1.bam.parts/_temporary/0/_temporary/attempt_202510271737505759150451256143092_0132_r_000000_0 dst=/user/runner/ReadsSparkSinkUnitTest1_bca83413-d6b6-4abe-a6f6-e11360f734b1.bam.parts/_temporary/0/task_202510271737505759150451256143092_0132_r_000000 perm=runner:supergroup:rwxr-xr-x proto=rpc
17:37:51.190 INFO FileOutputCommitter - Saved output of task 'attempt_202510271737505759150451256143092_0132_r_000000_0' to hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest1_bca83413-d6b6-4abe-a6f6-e11360f734b1.bam.parts/_temporary/0/task_202510271737505759150451256143092_0132_r_000000
17:37:51.190 INFO SparkHadoopMapRedUtil - attempt_202510271737505759150451256143092_0132_r_000000_0: Committed. Elapsed time: 2 ms.
17:37:51.191 INFO Executor - Finished task 0.0 in stage 40.0 (TID 78). 1858 bytes result sent to driver
17:37:51.192 INFO TaskSetManager - Finished task 0.0 in stage 40.0 (TID 78) in 457 ms on localhost (executor driver) (1/1)
17:37:51.192 INFO TaskSchedulerImpl - Removed TaskSet 40.0, whose tasks have all completed, from pool
17:37:51.192 INFO DAGScheduler - ResultStage 40 (runJob at SparkHadoopWriter.scala:83) finished in 0.466 s
17:37:51.193 INFO DAGScheduler - Job 28 is finished. Cancelling potential speculative or zombie tasks for this job
17:37:51.193 INFO TaskSchedulerImpl - Killing all running tasks in stage 40: Stage finished
17:37:51.193 INFO DAGScheduler - Job 28 finished: runJob at SparkHadoopWriter.scala:83, took 0.550211 s
17:37:51.193 INFO SparkHadoopWriter - Start to commit write Job job_202510271737505759150451256143092_0132.
17:37:51.194 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=listStatus src=/user/runner/ReadsSparkSinkUnitTest1_bca83413-d6b6-4abe-a6f6-e11360f734b1.bam.parts/_temporary/0 dst=null perm=null proto=rpc
17:37:51.195 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_bca83413-d6b6-4abe-a6f6-e11360f734b1.bam.parts dst=null perm=null proto=rpc
17:37:51.195 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=listStatus src=/user/runner/ReadsSparkSinkUnitTest1_bca83413-d6b6-4abe-a6f6-e11360f734b1.bam.parts/_temporary/0/task_202510271737505759150451256143092_0132_r_000000 dst=null perm=null proto=rpc
17:37:51.196 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_bca83413-d6b6-4abe-a6f6-e11360f734b1.bam.parts/.part-r-00000.bai dst=null perm=null proto=rpc
17:37:51.197 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=rename src=/user/runner/ReadsSparkSinkUnitTest1_bca83413-d6b6-4abe-a6f6-e11360f734b1.bam.parts/_temporary/0/task_202510271737505759150451256143092_0132_r_000000/.part-r-00000.bai dst=/user/runner/ReadsSparkSinkUnitTest1_bca83413-d6b6-4abe-a6f6-e11360f734b1.bam.parts/.part-r-00000.bai perm=runner:supergroup:rw-r--r-- proto=rpc
17:37:51.198 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_bca83413-d6b6-4abe-a6f6-e11360f734b1.bam.parts/part-r-00000 dst=null perm=null proto=rpc
17:37:51.198 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=rename src=/user/runner/ReadsSparkSinkUnitTest1_bca83413-d6b6-4abe-a6f6-e11360f734b1.bam.parts/_temporary/0/task_202510271737505759150451256143092_0132_r_000000/part-r-00000 dst=/user/runner/ReadsSparkSinkUnitTest1_bca83413-d6b6-4abe-a6f6-e11360f734b1.bam.parts/part-r-00000 perm=runner:supergroup:rw-r--r-- proto=rpc
17:37:51.199 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=delete src=/user/runner/ReadsSparkSinkUnitTest1_bca83413-d6b6-4abe-a6f6-e11360f734b1.bam.parts/_temporary dst=null perm=null proto=rpc
17:37:51.200 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest1_bca83413-d6b6-4abe-a6f6-e11360f734b1.bam.parts/_SUCCESS dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:37:51.201 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest1_bca83413-d6b6-4abe-a6f6-e11360f734b1.bam.parts/_SUCCESS is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:37:51.202 INFO audit - allowed=false ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=delete src=/user/runner/ReadsSparkSinkUnitTest1_bca83413-d6b6-4abe-a6f6-e11360f734b1.bam.parts/.spark-staging-132 dst=null perm=null proto=rpc
17:37:51.202 INFO SparkHadoopWriter - Write Job job_202510271737505759150451256143092_0132 committed. Elapsed time: 8 ms.
17:37:51.203 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest1_bca83413-d6b6-4abe-a6f6-e11360f734b1.bam.parts/header dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:37:51.203 WARN IntelDeflaterFactory - Intel Deflater not supported, using Java.util.zip.Deflater
17:37:51.205 INFO StateChange - BLOCK* allocate blk_1073741841_1017, replicas=127.0.0.1:37927 for /user/runner/ReadsSparkSinkUnitTest1_bca83413-d6b6-4abe-a6f6-e11360f734b1.bam.parts/header
17:37:51.206 INFO DataNode - Receiving BP-617366194-10.1.0.252-1761586664431:blk_1073741841_1017 src: /127.0.0.1:56608 dest: /127.0.0.1:37927
17:37:51.207 INFO clienttrace - src: /127.0.0.1:56608, dest: /127.0.0.1:37927, bytes: 5901, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1369393814_1, offset: 0, srvID: 379dd5e5-59c3-42cd-b9d3-67ca5e44560b, blockid: BP-617366194-10.1.0.252-1761586664431:blk_1073741841_1017, duration(ns): 553217
17:37:51.207 INFO DataNode - PacketResponder: BP-617366194-10.1.0.252-1761586664431:blk_1073741841_1017, type=LAST_IN_PIPELINE terminating
17:37:51.208 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest1_bca83413-d6b6-4abe-a6f6-e11360f734b1.bam.parts/header is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:37:51.209 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest1_bca83413-d6b6-4abe-a6f6-e11360f734b1.bam.parts/terminator dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:37:51.210 INFO StateChange - BLOCK* allocate blk_1073741842_1018, replicas=127.0.0.1:37927 for /user/runner/ReadsSparkSinkUnitTest1_bca83413-d6b6-4abe-a6f6-e11360f734b1.bam.parts/terminator
17:37:51.211 INFO DataNode - Receiving BP-617366194-10.1.0.252-1761586664431:blk_1073741842_1018 src: /127.0.0.1:56614 dest: /127.0.0.1:37927
17:37:51.212 INFO clienttrace - src: /127.0.0.1:56614, dest: /127.0.0.1:37927, bytes: 28, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1369393814_1, offset: 0, srvID: 379dd5e5-59c3-42cd-b9d3-67ca5e44560b, blockid: BP-617366194-10.1.0.252-1761586664431:blk_1073741842_1018, duration(ns): 558778
17:37:51.212 INFO DataNode - PacketResponder: BP-617366194-10.1.0.252-1761586664431:blk_1073741842_1018, type=LAST_IN_PIPELINE terminating
17:37:51.213 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest1_bca83413-d6b6-4abe-a6f6-e11360f734b1.bam.parts/terminator is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:37:51.214 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=listStatus src=/user/runner/ReadsSparkSinkUnitTest1_bca83413-d6b6-4abe-a6f6-e11360f734b1.bam.parts dst=null perm=null proto=rpc
17:37:51.215 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest1_bca83413-d6b6-4abe-a6f6-e11360f734b1.bam.parts/output dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:37:51.216 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest1_bca83413-d6b6-4abe-a6f6-e11360f734b1.bam.parts/output is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:37:51.216 INFO HadoopFileSystemWrapper - Concatenating 3 parts to hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest1_bca83413-d6b6-4abe-a6f6-e11360f734b1.bam
17:37:51.217 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=concat src=[/user/runner/ReadsSparkSinkUnitTest1_bca83413-d6b6-4abe-a6f6-e11360f734b1.bam.parts/header, /user/runner/ReadsSparkSinkUnitTest1_bca83413-d6b6-4abe-a6f6-e11360f734b1.bam.parts/part-r-00000, /user/runner/ReadsSparkSinkUnitTest1_bca83413-d6b6-4abe-a6f6-e11360f734b1.bam.parts/terminator] dst=/user/runner/ReadsSparkSinkUnitTest1_bca83413-d6b6-4abe-a6f6-e11360f734b1.bam.parts/output perm=runner:supergroup:rw-r--r-- proto=rpc
17:37:51.217 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_bca83413-d6b6-4abe-a6f6-e11360f734b1.bam dst=null perm=null proto=rpc
17:37:51.218 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=rename src=/user/runner/ReadsSparkSinkUnitTest1_bca83413-d6b6-4abe-a6f6-e11360f734b1.bam.parts/output dst=/user/runner/ReadsSparkSinkUnitTest1_bca83413-d6b6-4abe-a6f6-e11360f734b1.bam perm=runner:supergroup:rw-r--r-- proto=rpc
17:37:51.218 INFO HadoopFileSystemWrapper - Concatenating to hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest1_bca83413-d6b6-4abe-a6f6-e11360f734b1.bam done
17:37:51.219 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_bca83413-d6b6-4abe-a6f6-e11360f734b1.bam dst=null perm=null proto=rpc
17:37:51.219 INFO IndexFileMerger - Merging .bai files in temp directory hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest1_bca83413-d6b6-4abe-a6f6-e11360f734b1.bam.parts/ to hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest1_bca83413-d6b6-4abe-a6f6-e11360f734b1.bam.bai
17:37:51.219 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=listStatus src=/user/runner/ReadsSparkSinkUnitTest1_bca83413-d6b6-4abe-a6f6-e11360f734b1.bam.parts dst=null perm=null proto=rpc
17:37:51.220 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest1_bca83413-d6b6-4abe-a6f6-e11360f734b1.bam.bai dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:37:51.221 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_bca83413-d6b6-4abe-a6f6-e11360f734b1.bam.parts/.part-r-00000.bai dst=null perm=null proto=rpc
17:37:51.222 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_bca83413-d6b6-4abe-a6f6-e11360f734b1.bam.parts/.part-r-00000.bai dst=null perm=null proto=rpc
17:37:51.223 WARN DFSUtil - Unexpected value for data transfer bytes=5516 duration=0
17:37:51.224 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=delete src=/user/runner/ReadsSparkSinkUnitTest1_bca83413-d6b6-4abe-a6f6-e11360f734b1.bam.parts/.part-r-00000.bai dst=null perm=null proto=rpc
17:37:51.227 INFO StateChange - BLOCK* allocate blk_1073741843_1019, replicas=127.0.0.1:37927 for /user/runner/ReadsSparkSinkUnitTest1_bca83413-d6b6-4abe-a6f6-e11360f734b1.bam.bai
17:37:51.228 INFO DataNode - Receiving BP-617366194-10.1.0.252-1761586664431:blk_1073741843_1019 src: /127.0.0.1:56628 dest: /127.0.0.1:37927
17:37:51.230 INFO clienttrace - src: /127.0.0.1:56628, dest: /127.0.0.1:37927, bytes: 5472, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1369393814_1, offset: 0, srvID: 379dd5e5-59c3-42cd-b9d3-67ca5e44560b, blockid: BP-617366194-10.1.0.252-1761586664431:blk_1073741843_1019, duration(ns): 574009
17:37:51.230 INFO DataNode - PacketResponder: BP-617366194-10.1.0.252-1761586664431:blk_1073741843_1019, type=LAST_IN_PIPELINE terminating
17:37:51.230 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest1_bca83413-d6b6-4abe-a6f6-e11360f734b1.bam.bai is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:37:51.231 INFO IndexFileMerger - Done merging .bai files
17:37:51.231 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=delete src=/user/runner/ReadsSparkSinkUnitTest1_bca83413-d6b6-4abe-a6f6-e11360f734b1.bam.parts dst=null perm=null proto=rpc
17:37:51.240 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_bca83413-d6b6-4abe-a6f6-e11360f734b1.bam.bai dst=null perm=null proto=rpc
17:37:51.241 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_bca83413-d6b6-4abe-a6f6-e11360f734b1.bam dst=null perm=null proto=rpc
17:37:51.242 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_bca83413-d6b6-4abe-a6f6-e11360f734b1.bam dst=null perm=null proto=rpc
17:37:51.242 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_bca83413-d6b6-4abe-a6f6-e11360f734b1.bam dst=null perm=null proto=rpc
17:37:51.243 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_bca83413-d6b6-4abe-a6f6-e11360f734b1.bam dst=null perm=null proto=rpc
17:37:51.244 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_bca83413-d6b6-4abe-a6f6-e11360f734b1.bam.bai dst=null perm=null proto=rpc
17:37:51.245 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_bca83413-d6b6-4abe-a6f6-e11360f734b1.bam.bai dst=null perm=null proto=rpc
17:37:51.246 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_bca83413-d6b6-4abe-a6f6-e11360f734b1.bam.bai dst=null perm=null proto=rpc
17:37:51.247 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:37:51.247 WARN DFSUtil - Unexpected value for data transfer bytes=5949 duration=0
17:37:51.250 WARN DFSUtil - Unexpected value for data transfer bytes=5516 duration=0
17:37:51.251 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:51.251 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_bca83413-d6b6-4abe-a6f6-e11360f734b1.bam.sbi dst=null perm=null proto=rpc
17:37:51.253 INFO MemoryStore - Block broadcast_62 stored as values in memory (estimated size 297.9 KiB, free 1916.1 MiB)
17:37:51.261 INFO BlockManagerInfo - Removed broadcast_52_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.3 MiB)
17:37:51.262 INFO BlockManagerInfo - Removed broadcast_54_piece0 on localhost:37953 in memory (size: 153.6 KiB, free: 1919.5 MiB)
17:37:51.263 INFO BlockManagerInfo - Removed broadcast_51_piece0 on localhost:37953 in memory (size: 8.3 KiB, free: 1919.5 MiB)
17:37:51.264 INFO BlockManagerInfo - Removed broadcast_45_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.5 MiB)
17:37:51.264 INFO MemoryStore - Block broadcast_62_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1917.3 MiB)
17:37:51.265 INFO BlockManagerInfo - Added broadcast_62_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.5 MiB)
17:37:51.265 INFO BlockManagerInfo - Removed broadcast_61_piece0 on localhost:37953 in memory (size: 67.1 KiB, free: 1919.6 MiB)
17:37:51.265 INFO SparkContext - Created broadcast 62 from newAPIHadoopFile at PathSplitSource.java:96
17:37:51.266 INFO BlockManagerInfo - Removed broadcast_58_piece0 on localhost:37953 in memory (size: 9.6 KiB, free: 1919.6 MiB)
17:37:51.267 INFO BlockManagerInfo - Removed broadcast_59_piece0 on localhost:37953 in memory (size: 9.6 KiB, free: 1919.6 MiB)
17:37:51.268 INFO BlockManagerInfo - Removed broadcast_57_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.6 MiB)
17:37:51.269 INFO BlockManagerInfo - Removed broadcast_60_piece0 on localhost:37953 in memory (size: 166.1 KiB, free: 1919.8 MiB)
17:37:51.270 INFO BlockManagerInfo - Removed broadcast_55_piece0 on localhost:37953 in memory (size: 54.6 KiB, free: 1919.8 MiB)
17:37:51.271 INFO BlockManagerInfo - Removed broadcast_53_piece0 on localhost:37953 in memory (size: 54.6 KiB, free: 1919.9 MiB)
17:37:51.287 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_bca83413-d6b6-4abe-a6f6-e11360f734b1.bam dst=null perm=null proto=rpc
17:37:51.288 INFO FileInputFormat - Total input files to process : 1
17:37:51.288 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_bca83413-d6b6-4abe-a6f6-e11360f734b1.bam dst=null perm=null proto=rpc
17:37:51.322 INFO SparkContext - Starting job: collect at ReadsSparkSinkUnitTest.java:182
17:37:51.323 INFO DAGScheduler - Got job 29 (collect at ReadsSparkSinkUnitTest.java:182) with 1 output partitions
17:37:51.323 INFO DAGScheduler - Final stage: ResultStage 41 (collect at ReadsSparkSinkUnitTest.java:182)
17:37:51.323 INFO DAGScheduler - Parents of final stage: List()
17:37:51.323 INFO DAGScheduler - Missing parents: List()
17:37:51.323 INFO DAGScheduler - Submitting ResultStage 41 (MapPartitionsRDD[139] at filter at ReadsSparkSource.java:96), which has no missing parents
17:37:51.339 INFO MemoryStore - Block broadcast_63 stored as values in memory (estimated size 426.2 KiB, free 1918.9 MiB)
17:37:51.340 INFO MemoryStore - Block broadcast_63_piece0 stored as bytes in memory (estimated size 153.7 KiB, free 1918.8 MiB)
17:37:51.340 INFO BlockManagerInfo - Added broadcast_63_piece0 in memory on localhost:37953 (size: 153.7 KiB, free: 1919.8 MiB)
17:37:51.341 INFO SparkContext - Created broadcast 63 from broadcast at DAGScheduler.scala:1580
17:37:51.341 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 41 (MapPartitionsRDD[139] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:37:51.341 INFO TaskSchedulerImpl - Adding task set 41.0 with 1 tasks resource profile 0
17:37:51.342 INFO TaskSetManager - Starting task 0.0 in stage 41.0 (TID 79) (localhost, executor driver, partition 0, ANY, 7852 bytes)
17:37:51.342 INFO Executor - Running task 0.0 in stage 41.0 (TID 79)
17:37:51.376 INFO NewHadoopRDD - Input split: hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest1_bca83413-d6b6-4abe-a6f6-e11360f734b1.bam:0+238969
17:37:51.377 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_bca83413-d6b6-4abe-a6f6-e11360f734b1.bam dst=null perm=null proto=rpc
17:37:51.378 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_bca83413-d6b6-4abe-a6f6-e11360f734b1.bam dst=null perm=null proto=rpc
17:37:51.380 WARN DFSUtil - Unexpected value for data transfer bytes=5949 duration=0
17:37:51.381 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_bca83413-d6b6-4abe-a6f6-e11360f734b1.bam dst=null perm=null proto=rpc
17:37:51.381 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_bca83413-d6b6-4abe-a6f6-e11360f734b1.bam dst=null perm=null proto=rpc
17:37:51.382 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_bca83413-d6b6-4abe-a6f6-e11360f734b1.bam.bai dst=null perm=null proto=rpc
17:37:51.383 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_bca83413-d6b6-4abe-a6f6-e11360f734b1.bam.bai dst=null perm=null proto=rpc
17:37:51.384 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_bca83413-d6b6-4abe-a6f6-e11360f734b1.bam.bai dst=null perm=null proto=rpc
17:37:51.385 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:37:51.386 WARN DFSUtil - Unexpected value for data transfer bytes=5949 duration=0
17:37:51.389 WARN DFSUtil - Unexpected value for data transfer bytes=5516 duration=0
17:37:51.390 WARN DFSUtil - Unexpected value for data transfer bytes=5516 duration=0
17:37:51.390 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_bca83413-d6b6-4abe-a6f6-e11360f734b1.bam dst=null perm=null proto=rpc
17:37:51.391 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_bca83413-d6b6-4abe-a6f6-e11360f734b1.bam dst=null perm=null proto=rpc
17:37:51.392 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:37:51.392 WARN DFSUtil - Unexpected value for data transfer bytes=5949 duration=0
17:37:51.397 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:51.398 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:51.399 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:51.400 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:51.403 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:51.403 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:51.404 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:51.405 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:51.406 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:51.407 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:51.408 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:51.409 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:51.411 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:51.412 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:51.414 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:51.415 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:51.417 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:51.418 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:51.419 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:51.421 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:51.423 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:51.424 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:51.425 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:51.427 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:51.428 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:51.429 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:51.430 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:51.431 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:51.434 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:51.435 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:51.436 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:51.437 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:51.438 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:51.440 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:51.442 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:51.442 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:51.443 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:51.444 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:51.445 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:51.447 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:51.447 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:51.448 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:51.449 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:51.450 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:51.453 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:51.454 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:51.455 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:51.456 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:51.457 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:51.458 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:51.461 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:51.462 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:51.463 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_bca83413-d6b6-4abe-a6f6-e11360f734b1.bam dst=null perm=null proto=rpc
17:37:51.463 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_bca83413-d6b6-4abe-a6f6-e11360f734b1.bam dst=null perm=null proto=rpc
17:37:51.464 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_bca83413-d6b6-4abe-a6f6-e11360f734b1.bam.bai dst=null perm=null proto=rpc
17:37:51.465 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_bca83413-d6b6-4abe-a6f6-e11360f734b1.bam.bai dst=null perm=null proto=rpc
17:37:51.466 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_bca83413-d6b6-4abe-a6f6-e11360f734b1.bam.bai dst=null perm=null proto=rpc
17:37:51.467 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:37:51.468 WARN DFSUtil - Unexpected value for data transfer bytes=5949 duration=0
17:37:51.470 WARN DFSUtil - Unexpected value for data transfer bytes=5516 duration=0
17:37:51.472 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:51.473 WARN DFSUtil - Unexpected value for data transfer bytes=32 duration=0
17:37:51.477 INFO Executor - Finished task 0.0 in stage 41.0 (TID 79). 651526 bytes result sent to driver
17:37:51.479 INFO TaskSetManager - Finished task 0.0 in stage 41.0 (TID 79) in 137 ms on localhost (executor driver) (1/1)
17:37:51.479 INFO TaskSchedulerImpl - Removed TaskSet 41.0, whose tasks have all completed, from pool
17:37:51.479 INFO DAGScheduler - ResultStage 41 (collect at ReadsSparkSinkUnitTest.java:182) finished in 0.156 s
17:37:51.480 INFO DAGScheduler - Job 29 is finished. Cancelling potential speculative or zombie tasks for this job
17:37:51.480 INFO TaskSchedulerImpl - Killing all running tasks in stage 41: Stage finished
17:37:51.480 INFO DAGScheduler - Job 29 finished: collect at ReadsSparkSinkUnitTest.java:182, took 0.158077 s
17:37:51.492 INFO SparkContext - Starting job: count at ReadsSparkSinkUnitTest.java:185
17:37:51.493 INFO DAGScheduler - Got job 30 (count at ReadsSparkSinkUnitTest.java:185) with 1 output partitions
17:37:51.493 INFO DAGScheduler - Final stage: ResultStage 42 (count at ReadsSparkSinkUnitTest.java:185)
17:37:51.493 INFO DAGScheduler - Parents of final stage: List()
17:37:51.493 INFO DAGScheduler - Missing parents: List()
17:37:51.493 INFO DAGScheduler - Submitting ResultStage 42 (MapPartitionsRDD[120] at filter at ReadsSparkSource.java:96), which has no missing parents
17:37:51.509 INFO MemoryStore - Block broadcast_64 stored as values in memory (estimated size 426.1 KiB, free 1918.3 MiB)
17:37:51.510 INFO MemoryStore - Block broadcast_64_piece0 stored as bytes in memory (estimated size 153.6 KiB, free 1918.2 MiB)
17:37:51.510 INFO BlockManagerInfo - Added broadcast_64_piece0 in memory on localhost:37953 (size: 153.6 KiB, free: 1919.6 MiB)
17:37:51.511 INFO SparkContext - Created broadcast 64 from broadcast at DAGScheduler.scala:1580
17:37:51.511 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 42 (MapPartitionsRDD[120] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:37:51.511 INFO TaskSchedulerImpl - Adding task set 42.0 with 1 tasks resource profile 0
17:37:51.512 INFO TaskSetManager - Starting task 0.0 in stage 42.0 (TID 80) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7872 bytes)
17:37:51.512 INFO Executor - Running task 0.0 in stage 42.0 (TID 80)
17:37:51.544 INFO NewHadoopRDD - Input split: file:/home/runner/work/gatk/gatk/src/test/resources/org/broadinstitute/hellbender/tools/BQSR/HiSeq.1mb.1RG.2k_lines.bam:0+222075
17:37:51.545 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:37:51.547 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:37:51.554 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:37:51.558 INFO Executor - Finished task 0.0 in stage 42.0 (TID 80). 989 bytes result sent to driver
17:37:51.559 INFO TaskSetManager - Finished task 0.0 in stage 42.0 (TID 80) in 48 ms on localhost (executor driver) (1/1)
17:37:51.559 INFO TaskSchedulerImpl - Removed TaskSet 42.0, whose tasks have all completed, from pool
17:37:51.559 INFO DAGScheduler - ResultStage 42 (count at ReadsSparkSinkUnitTest.java:185) finished in 0.066 s
17:37:51.559 INFO DAGScheduler - Job 30 is finished. Cancelling potential speculative or zombie tasks for this job
17:37:51.559 INFO TaskSchedulerImpl - Killing all running tasks in stage 42: Stage finished
17:37:51.559 INFO DAGScheduler - Job 30 finished: count at ReadsSparkSinkUnitTest.java:185, took 0.067001 s
17:37:51.562 INFO SparkContext - Starting job: count at ReadsSparkSinkUnitTest.java:185
17:37:51.563 INFO DAGScheduler - Got job 31 (count at ReadsSparkSinkUnitTest.java:185) with 1 output partitions
17:37:51.563 INFO DAGScheduler - Final stage: ResultStage 43 (count at ReadsSparkSinkUnitTest.java:185)
17:37:51.563 INFO DAGScheduler - Parents of final stage: List()
17:37:51.563 INFO DAGScheduler - Missing parents: List()
17:37:51.563 INFO DAGScheduler - Submitting ResultStage 43 (MapPartitionsRDD[139] at filter at ReadsSparkSource.java:96), which has no missing parents
17:37:51.579 INFO MemoryStore - Block broadcast_65 stored as values in memory (estimated size 426.1 KiB, free 1917.8 MiB)
17:37:51.580 INFO MemoryStore - Block broadcast_65_piece0 stored as bytes in memory (estimated size 153.6 KiB, free 1917.6 MiB)
17:37:51.581 INFO BlockManagerInfo - Added broadcast_65_piece0 in memory on localhost:37953 (size: 153.6 KiB, free: 1919.5 MiB)
17:37:51.582 INFO SparkContext - Created broadcast 65 from broadcast at DAGScheduler.scala:1580
17:37:51.582 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 43 (MapPartitionsRDD[139] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:37:51.582 INFO TaskSchedulerImpl - Adding task set 43.0 with 1 tasks resource profile 0
17:37:51.583 INFO TaskSetManager - Starting task 0.0 in stage 43.0 (TID 81) (localhost, executor driver, partition 0, ANY, 7852 bytes)
17:37:51.584 INFO Executor - Running task 0.0 in stage 43.0 (TID 81)
17:37:51.598 INFO FsDatasetAsyncDiskService - Scheduling blk_1073741827_1003 replica FinalizedReplica, blk_1073741827_1003, FINALIZED
getNumBytes() = 5472
getBytesOnDisk() = 5472
getVisibleLength()= 5472
getVolume() = /tmp/minicluster_storage14473397491642437130/data/data1
getBlockURI() = file:/tmp/minicluster_storage14473397491642437130/data/data1/current/BP-617366194-10.1.0.252-1761586664431/current/finalized/subdir0/subdir0/blk_1073741827 for deletion
17:37:51.598 INFO FsDatasetAsyncDiskService - Scheduling blk_1073741833_1009 replica FinalizedReplica, blk_1073741833_1009, FINALIZED
getNumBytes() = 13492
getBytesOnDisk() = 13492
getVisibleLength()= 13492
getVolume() = /tmp/minicluster_storage14473397491642437130/data/data1
getBlockURI() = file:/tmp/minicluster_storage14473397491642437130/data/data1/current/BP-617366194-10.1.0.252-1761586664431/current/finalized/subdir0/subdir0/blk_1073741833 for deletion
17:37:51.598 INFO FsDatasetAsyncDiskService - Scheduling blk_1073741834_1010 replica FinalizedReplica, blk_1073741834_1010, FINALIZED
getNumBytes() = 5472
getBytesOnDisk() = 5472
getVisibleLength()= 5472
getVolume() = /tmp/minicluster_storage14473397491642437130/data/data2
getBlockURI() = file:/tmp/minicluster_storage14473397491642437130/data/data2/current/BP-617366194-10.1.0.252-1761586664431/current/finalized/subdir0/subdir0/blk_1073741834 for deletion
17:37:51.598 INFO FsDatasetAsyncDiskService - Deleted BP-617366194-10.1.0.252-1761586664431 blk_1073741827_1003 URI file:/tmp/minicluster_storage14473397491642437130/data/data1/current/BP-617366194-10.1.0.252-1761586664431/current/finalized/subdir0/subdir0/blk_1073741827
17:37:51.599 INFO FsDatasetAsyncDiskService - Deleted BP-617366194-10.1.0.252-1761586664431 blk_1073741833_1009 URI file:/tmp/minicluster_storage14473397491642437130/data/data1/current/BP-617366194-10.1.0.252-1761586664431/current/finalized/subdir0/subdir0/blk_1073741833
17:37:51.599 INFO FsDatasetAsyncDiskService - Deleted BP-617366194-10.1.0.252-1761586664431 blk_1073741834_1010 URI file:/tmp/minicluster_storage14473397491642437130/data/data2/current/BP-617366194-10.1.0.252-1761586664431/current/finalized/subdir0/subdir0/blk_1073741834
17:37:51.615 INFO NewHadoopRDD - Input split: hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest1_bca83413-d6b6-4abe-a6f6-e11360f734b1.bam:0+238969
17:37:51.616 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_bca83413-d6b6-4abe-a6f6-e11360f734b1.bam dst=null perm=null proto=rpc
17:37:51.617 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_bca83413-d6b6-4abe-a6f6-e11360f734b1.bam dst=null perm=null proto=rpc
17:37:51.619 WARN DFSUtil - Unexpected value for data transfer bytes=5949 duration=0
17:37:51.619 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_bca83413-d6b6-4abe-a6f6-e11360f734b1.bam dst=null perm=null proto=rpc
17:37:51.620 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_bca83413-d6b6-4abe-a6f6-e11360f734b1.bam dst=null perm=null proto=rpc
17:37:51.621 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_bca83413-d6b6-4abe-a6f6-e11360f734b1.bam.bai dst=null perm=null proto=rpc
17:37:51.622 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_bca83413-d6b6-4abe-a6f6-e11360f734b1.bam.bai dst=null perm=null proto=rpc
17:37:51.623 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_bca83413-d6b6-4abe-a6f6-e11360f734b1.bam.bai dst=null perm=null proto=rpc
17:37:51.624 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:37:51.624 WARN DFSUtil - Unexpected value for data transfer bytes=5949 duration=0
17:37:51.627 WARN DFSUtil - Unexpected value for data transfer bytes=5516 duration=0
17:37:51.628 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_bca83413-d6b6-4abe-a6f6-e11360f734b1.bam dst=null perm=null proto=rpc
17:37:51.628 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_bca83413-d6b6-4abe-a6f6-e11360f734b1.bam dst=null perm=null proto=rpc
17:37:51.629 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:37:51.629 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:51.630 WARN DFSUtil - Unexpected value for data transfer bytes=5949 duration=0
17:37:51.634 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:51.635 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:51.636 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:51.637 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:51.639 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:51.640 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:51.640 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:51.641 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:51.642 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:51.643 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:51.644 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:51.645 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:51.645 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:51.646 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:51.647 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:51.648 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:51.649 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:51.651 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:51.652 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:51.653 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:51.654 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:51.655 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:51.656 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:51.656 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:51.657 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:51.658 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:51.659 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:51.660 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:51.661 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:51.662 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:51.663 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:51.665 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:51.665 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:51.667 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:51.668 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:51.670 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:51.670 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:51.671 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:51.672 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:51.674 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:51.675 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:51.676 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:51.677 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:51.678 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:51.679 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:51.680 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:51.681 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:51.683 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:51.684 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:51.686 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:51.687 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:51.688 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:51.689 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:51.690 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:51.690 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:51.691 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:51.692 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:51.693 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:51.694 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_bca83413-d6b6-4abe-a6f6-e11360f734b1.bam dst=null perm=null proto=rpc
17:37:51.694 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_bca83413-d6b6-4abe-a6f6-e11360f734b1.bam dst=null perm=null proto=rpc
17:37:51.696 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_bca83413-d6b6-4abe-a6f6-e11360f734b1.bam.bai dst=null perm=null proto=rpc
17:37:51.697 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_bca83413-d6b6-4abe-a6f6-e11360f734b1.bam.bai dst=null perm=null proto=rpc
17:37:51.698 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_bca83413-d6b6-4abe-a6f6-e11360f734b1.bam.bai dst=null perm=null proto=rpc
17:37:51.699 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:37:51.700 WARN DFSUtil - Unexpected value for data transfer bytes=5949 duration=0
17:37:51.702 WARN DFSUtil - Unexpected value for data transfer bytes=5516 duration=0
17:37:51.704 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:51.705 WARN DFSUtil - Unexpected value for data transfer bytes=32 duration=0
17:37:51.707 INFO Executor - Finished task 0.0 in stage 43.0 (TID 81). 989 bytes result sent to driver
17:37:51.708 INFO TaskSetManager - Finished task 0.0 in stage 43.0 (TID 81) in 125 ms on localhost (executor driver) (1/1)
17:37:51.708 INFO TaskSchedulerImpl - Removed TaskSet 43.0, whose tasks have all completed, from pool
17:37:51.708 INFO DAGScheduler - ResultStage 43 (count at ReadsSparkSinkUnitTest.java:185) finished in 0.144 s
17:37:51.708 INFO DAGScheduler - Job 31 is finished. Cancelling potential speculative or zombie tasks for this job
17:37:51.708 INFO TaskSchedulerImpl - Killing all running tasks in stage 43: Stage finished
17:37:51.708 INFO DAGScheduler - Job 31 finished: count at ReadsSparkSinkUnitTest.java:185, took 0.145729 s
17:37:51.709 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:37:51.712 INFO MemoryStore - Block broadcast_66 stored as values in memory (estimated size 297.9 KiB, free 1917.3 MiB)
17:37:51.718 INFO MemoryStore - Block broadcast_66_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1917.3 MiB)
17:37:51.718 INFO BlockManagerInfo - Added broadcast_66_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.4 MiB)
17:37:51.719 INFO SparkContext - Created broadcast 66 from newAPIHadoopFile at PathSplitSource.java:96
17:37:51.739 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:37:51.744 INFO BlockManagerInfo - Removed broadcast_63_piece0 on localhost:37953 in memory (size: 153.7 KiB, free: 1919.6 MiB)
17:37:51.745 INFO BlockManagerInfo - Removed broadcast_65_piece0 on localhost:37953 in memory (size: 153.6 KiB, free: 1919.7 MiB)
17:37:51.746 INFO MemoryStore - Block broadcast_67 stored as values in memory (estimated size 297.9 KiB, free 1917.7 MiB)
17:37:51.746 INFO BlockManagerInfo - Removed broadcast_62_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.8 MiB)
17:37:51.747 INFO BlockManagerInfo - Removed broadcast_56_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.8 MiB)
17:37:51.747 INFO BlockManagerInfo - Removed broadcast_64_piece0 on localhost:37953 in memory (size: 153.6 KiB, free: 1920.0 MiB)
17:37:51.751 INFO MemoryStore - Block broadcast_67_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1919.3 MiB)
17:37:51.751 INFO BlockManagerInfo - Added broadcast_67_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.9 MiB)
17:37:51.752 INFO SparkContext - Created broadcast 67 from newAPIHadoopFile at PathSplitSource.java:96
17:37:51.772 INFO FileInputFormat - Total input files to process : 1
17:37:51.774 INFO MemoryStore - Block broadcast_68 stored as values in memory (estimated size 160.7 KiB, free 1919.2 MiB)
17:37:51.775 INFO MemoryStore - Block broadcast_68_piece0 stored as bytes in memory (estimated size 9.6 KiB, free 1919.2 MiB)
17:37:51.775 INFO BlockManagerInfo - Added broadcast_68_piece0 in memory on localhost:37953 (size: 9.6 KiB, free: 1919.9 MiB)
17:37:51.776 INFO SparkContext - Created broadcast 68 from broadcast at ReadsSparkSink.java:133
17:37:51.776 WARN HtsjdkReadsRddStorage - Unrecognized write option: DISABLE
17:37:51.777 INFO MemoryStore - Block broadcast_69 stored as values in memory (estimated size 163.2 KiB, free 1919.0 MiB)
17:37:51.778 INFO MemoryStore - Block broadcast_69_piece0 stored as bytes in memory (estimated size 9.6 KiB, free 1919.0 MiB)
17:37:51.778 INFO BlockManagerInfo - Added broadcast_69_piece0 in memory on localhost:37953 (size: 9.6 KiB, free: 1919.9 MiB)
17:37:51.779 INFO SparkContext - Created broadcast 69 from broadcast at BamSink.java:76
17:37:51.781 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_5caa91ea-3dd8-4cc9-baa0-a6274b4a2a8b.bam.parts dst=null perm=null proto=rpc
17:37:51.782 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:37:51.782 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:37:51.782 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:37:51.783 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=mkdirs src=/user/runner/ReadsSparkSinkUnitTest1_5caa91ea-3dd8-4cc9-baa0-a6274b4a2a8b.bam.parts/_temporary/0 dst=null perm=runner:supergroup:rwxr-xr-x proto=rpc
17:37:51.789 INFO SparkContext - Starting job: runJob at SparkHadoopWriter.scala:83
17:37:51.789 INFO DAGScheduler - Registering RDD 153 (mapToPair at SparkUtils.java:161) as input to shuffle 10
17:37:51.790 INFO DAGScheduler - Got job 32 (runJob at SparkHadoopWriter.scala:83) with 1 output partitions
17:37:51.790 INFO DAGScheduler - Final stage: ResultStage 45 (runJob at SparkHadoopWriter.scala:83)
17:37:51.790 INFO DAGScheduler - Parents of final stage: List(ShuffleMapStage 44)
17:37:51.790 INFO DAGScheduler - Missing parents: List(ShuffleMapStage 44)
17:37:51.790 INFO DAGScheduler - Submitting ShuffleMapStage 44 (MapPartitionsRDD[153] at mapToPair at SparkUtils.java:161), which has no missing parents
17:37:51.806 INFO MemoryStore - Block broadcast_70 stored as values in memory (estimated size 520.4 KiB, free 1918.5 MiB)
17:37:51.808 INFO MemoryStore - Block broadcast_70_piece0 stored as bytes in memory (estimated size 166.1 KiB, free 1918.3 MiB)
17:37:51.808 INFO BlockManagerInfo - Added broadcast_70_piece0 in memory on localhost:37953 (size: 166.1 KiB, free: 1919.7 MiB)
17:37:51.808 INFO SparkContext - Created broadcast 70 from broadcast at DAGScheduler.scala:1580
17:37:51.808 INFO DAGScheduler - Submitting 1 missing tasks from ShuffleMapStage 44 (MapPartitionsRDD[153] at mapToPair at SparkUtils.java:161) (first 15 tasks are for partitions Vector(0))
17:37:51.808 INFO TaskSchedulerImpl - Adding task set 44.0 with 1 tasks resource profile 0
17:37:51.809 INFO TaskSetManager - Starting task 0.0 in stage 44.0 (TID 82) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7861 bytes)
17:37:51.810 INFO Executor - Running task 0.0 in stage 44.0 (TID 82)
17:37:51.845 INFO NewHadoopRDD - Input split: file:/home/runner/work/gatk/gatk/src/test/resources/org/broadinstitute/hellbender/tools/BQSR/HiSeq.1mb.1RG.2k_lines.bam:0+222075
17:37:51.846 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:37:51.847 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:37:51.854 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:37:51.865 INFO Executor - Finished task 0.0 in stage 44.0 (TID 82). 1148 bytes result sent to driver
17:37:51.866 INFO TaskSetManager - Finished task 0.0 in stage 44.0 (TID 82) in 57 ms on localhost (executor driver) (1/1)
17:37:51.866 INFO TaskSchedulerImpl - Removed TaskSet 44.0, whose tasks have all completed, from pool
17:37:51.866 INFO DAGScheduler - ShuffleMapStage 44 (mapToPair at SparkUtils.java:161) finished in 0.076 s
17:37:51.866 INFO DAGScheduler - looking for newly runnable stages
17:37:51.866 INFO DAGScheduler - running: HashSet()
17:37:51.866 INFO DAGScheduler - waiting: HashSet(ResultStage 45)
17:37:51.866 INFO DAGScheduler - failed: HashSet()
17:37:51.866 INFO DAGScheduler - Submitting ResultStage 45 (MapPartitionsRDD[158] at mapToPair at BamSink.java:91), which has no missing parents
17:37:51.873 INFO MemoryStore - Block broadcast_71 stored as values in memory (estimated size 241.5 KiB, free 1918.1 MiB)
17:37:51.874 INFO MemoryStore - Block broadcast_71_piece0 stored as bytes in memory (estimated size 67.1 KiB, free 1918.0 MiB)
17:37:51.874 INFO BlockManagerInfo - Added broadcast_71_piece0 in memory on localhost:37953 (size: 67.1 KiB, free: 1919.7 MiB)
17:37:51.874 INFO SparkContext - Created broadcast 71 from broadcast at DAGScheduler.scala:1580
17:37:51.874 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 45 (MapPartitionsRDD[158] at mapToPair at BamSink.java:91) (first 15 tasks are for partitions Vector(0))
17:37:51.874 INFO TaskSchedulerImpl - Adding task set 45.0 with 1 tasks resource profile 0
17:37:51.875 INFO TaskSetManager - Starting task 0.0 in stage 45.0 (TID 83) (localhost, executor driver, partition 0, NODE_LOCAL, 7513 bytes)
17:37:51.875 INFO Executor - Running task 0.0 in stage 45.0 (TID 83)
17:37:51.881 INFO ShuffleBlockFetcherIterator - Getting 1 (343.8 KiB) non-empty blocks including 1 (343.8 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
17:37:51.881 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
17:37:51.897 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:37:51.897 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:37:51.897 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:37:51.898 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:37:51.898 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:37:51.898 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:37:51.899 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest1_5caa91ea-3dd8-4cc9-baa0-a6274b4a2a8b.bam.parts/_temporary/0/_temporary/attempt_202510271737517782725299037987949_0158_r_000000_0/part-r-00000 dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:37:51.900 WARN IntelDeflaterFactory - Intel Deflater not supported, using Java.util.zip.Deflater
17:37:51.900 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest1_5caa91ea-3dd8-4cc9-baa0-a6274b4a2a8b.bam.parts/_temporary/0/_temporary/attempt_202510271737517782725299037987949_0158_r_000000_0/.part-r-00000.sbi dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:37:51.905 INFO StateChange - BLOCK* allocate blk_1073741844_1020, replicas=127.0.0.1:37927 for /user/runner/ReadsSparkSinkUnitTest1_5caa91ea-3dd8-4cc9-baa0-a6274b4a2a8b.bam.parts/_temporary/0/_temporary/attempt_202510271737517782725299037987949_0158_r_000000_0/part-r-00000
17:37:51.906 INFO DataNode - Receiving BP-617366194-10.1.0.252-1761586664431:blk_1073741844_1020 src: /127.0.0.1:57290 dest: /127.0.0.1:37927
17:37:51.911 INFO clienttrace - src: /127.0.0.1:57290, dest: /127.0.0.1:37927, bytes: 233040, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1369393814_1, offset: 0, srvID: 379dd5e5-59c3-42cd-b9d3-67ca5e44560b, blockid: BP-617366194-10.1.0.252-1761586664431:blk_1073741844_1020, duration(ns): 4086605
17:37:51.911 INFO DataNode - PacketResponder: BP-617366194-10.1.0.252-1761586664431:blk_1073741844_1020, type=LAST_IN_PIPELINE terminating
17:37:51.912 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest1_5caa91ea-3dd8-4cc9-baa0-a6274b4a2a8b.bam.parts/_temporary/0/_temporary/attempt_202510271737517782725299037987949_0158_r_000000_0/part-r-00000 is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:37:51.913 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_5caa91ea-3dd8-4cc9-baa0-a6274b4a2a8b.bam.parts/_temporary/0/_temporary/attempt_202510271737517782725299037987949_0158_r_000000_0/part-r-00000 dst=null perm=null proto=rpc
17:37:51.914 INFO StateChange - BLOCK* allocate blk_1073741845_1021, replicas=127.0.0.1:37927 for /user/runner/ReadsSparkSinkUnitTest1_5caa91ea-3dd8-4cc9-baa0-a6274b4a2a8b.bam.parts/_temporary/0/_temporary/attempt_202510271737517782725299037987949_0158_r_000000_0/.part-r-00000.sbi
17:37:51.915 INFO DataNode - Receiving BP-617366194-10.1.0.252-1761586664431:blk_1073741845_1021 src: /127.0.0.1:57292 dest: /127.0.0.1:37927
17:37:51.916 INFO clienttrace - src: /127.0.0.1:57292, dest: /127.0.0.1:37927, bytes: 212, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1369393814_1, offset: 0, srvID: 379dd5e5-59c3-42cd-b9d3-67ca5e44560b, blockid: BP-617366194-10.1.0.252-1761586664431:blk_1073741845_1021, duration(ns): 490935
17:37:51.916 INFO DataNode - PacketResponder: BP-617366194-10.1.0.252-1761586664431:blk_1073741845_1021, type=LAST_IN_PIPELINE terminating
17:37:51.917 INFO FSNamesystem - BLOCK* blk_1073741845_1021 is COMMITTED but not COMPLETE(numNodes= 0 < minimum = 1) in file /user/runner/ReadsSparkSinkUnitTest1_5caa91ea-3dd8-4cc9-baa0-a6274b4a2a8b.bam.parts/_temporary/0/_temporary/attempt_202510271737517782725299037987949_0158_r_000000_0/.part-r-00000.sbi
17:37:52.318 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest1_5caa91ea-3dd8-4cc9-baa0-a6274b4a2a8b.bam.parts/_temporary/0/_temporary/attempt_202510271737517782725299037987949_0158_r_000000_0/.part-r-00000.sbi is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:37:52.319 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_5caa91ea-3dd8-4cc9-baa0-a6274b4a2a8b.bam.parts/_temporary/0/_temporary/attempt_202510271737517782725299037987949_0158_r_000000_0 dst=null perm=null proto=rpc
17:37:52.320 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_5caa91ea-3dd8-4cc9-baa0-a6274b4a2a8b.bam.parts/_temporary/0/_temporary/attempt_202510271737517782725299037987949_0158_r_000000_0 dst=null perm=null proto=rpc
17:37:52.321 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_5caa91ea-3dd8-4cc9-baa0-a6274b4a2a8b.bam.parts/_temporary/0/task_202510271737517782725299037987949_0158_r_000000 dst=null perm=null proto=rpc
17:37:52.321 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=rename src=/user/runner/ReadsSparkSinkUnitTest1_5caa91ea-3dd8-4cc9-baa0-a6274b4a2a8b.bam.parts/_temporary/0/_temporary/attempt_202510271737517782725299037987949_0158_r_000000_0 dst=/user/runner/ReadsSparkSinkUnitTest1_5caa91ea-3dd8-4cc9-baa0-a6274b4a2a8b.bam.parts/_temporary/0/task_202510271737517782725299037987949_0158_r_000000 perm=runner:supergroup:rwxr-xr-x proto=rpc
17:37:52.322 INFO FileOutputCommitter - Saved output of task 'attempt_202510271737517782725299037987949_0158_r_000000_0' to hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest1_5caa91ea-3dd8-4cc9-baa0-a6274b4a2a8b.bam.parts/_temporary/0/task_202510271737517782725299037987949_0158_r_000000
17:37:52.322 INFO SparkHadoopMapRedUtil - attempt_202510271737517782725299037987949_0158_r_000000_0: Committed. Elapsed time: 2 ms.
17:37:52.323 INFO Executor - Finished task 0.0 in stage 45.0 (TID 83). 1858 bytes result sent to driver
17:37:52.323 INFO TaskSetManager - Finished task 0.0 in stage 45.0 (TID 83) in 448 ms on localhost (executor driver) (1/1)
17:37:52.323 INFO TaskSchedulerImpl - Removed TaskSet 45.0, whose tasks have all completed, from pool
17:37:52.324 INFO DAGScheduler - ResultStage 45 (runJob at SparkHadoopWriter.scala:83) finished in 0.457 s
17:37:52.324 INFO DAGScheduler - Job 32 is finished. Cancelling potential speculative or zombie tasks for this job
17:37:52.324 INFO TaskSchedulerImpl - Killing all running tasks in stage 45: Stage finished
17:37:52.324 INFO DAGScheduler - Job 32 finished: runJob at SparkHadoopWriter.scala:83, took 0.535237 s
17:37:52.325 INFO SparkHadoopWriter - Start to commit write Job job_202510271737517782725299037987949_0158.
17:37:52.325 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=listStatus src=/user/runner/ReadsSparkSinkUnitTest1_5caa91ea-3dd8-4cc9-baa0-a6274b4a2a8b.bam.parts/_temporary/0 dst=null perm=null proto=rpc
17:37:52.326 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_5caa91ea-3dd8-4cc9-baa0-a6274b4a2a8b.bam.parts dst=null perm=null proto=rpc
17:37:52.327 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=listStatus src=/user/runner/ReadsSparkSinkUnitTest1_5caa91ea-3dd8-4cc9-baa0-a6274b4a2a8b.bam.parts/_temporary/0/task_202510271737517782725299037987949_0158_r_000000 dst=null perm=null proto=rpc
17:37:52.327 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_5caa91ea-3dd8-4cc9-baa0-a6274b4a2a8b.bam.parts/.part-r-00000.sbi dst=null perm=null proto=rpc
17:37:52.329 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=rename src=/user/runner/ReadsSparkSinkUnitTest1_5caa91ea-3dd8-4cc9-baa0-a6274b4a2a8b.bam.parts/_temporary/0/task_202510271737517782725299037987949_0158_r_000000/.part-r-00000.sbi dst=/user/runner/ReadsSparkSinkUnitTest1_5caa91ea-3dd8-4cc9-baa0-a6274b4a2a8b.bam.parts/.part-r-00000.sbi perm=runner:supergroup:rw-r--r-- proto=rpc
17:37:52.329 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_5caa91ea-3dd8-4cc9-baa0-a6274b4a2a8b.bam.parts/part-r-00000 dst=null perm=null proto=rpc
17:37:52.330 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=rename src=/user/runner/ReadsSparkSinkUnitTest1_5caa91ea-3dd8-4cc9-baa0-a6274b4a2a8b.bam.parts/_temporary/0/task_202510271737517782725299037987949_0158_r_000000/part-r-00000 dst=/user/runner/ReadsSparkSinkUnitTest1_5caa91ea-3dd8-4cc9-baa0-a6274b4a2a8b.bam.parts/part-r-00000 perm=runner:supergroup:rw-r--r-- proto=rpc
17:37:52.331 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=delete src=/user/runner/ReadsSparkSinkUnitTest1_5caa91ea-3dd8-4cc9-baa0-a6274b4a2a8b.bam.parts/_temporary dst=null perm=null proto=rpc
17:37:52.331 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest1_5caa91ea-3dd8-4cc9-baa0-a6274b4a2a8b.bam.parts/_SUCCESS dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:37:52.332 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest1_5caa91ea-3dd8-4cc9-baa0-a6274b4a2a8b.bam.parts/_SUCCESS is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:37:52.333 INFO audit - allowed=false ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=delete src=/user/runner/ReadsSparkSinkUnitTest1_5caa91ea-3dd8-4cc9-baa0-a6274b4a2a8b.bam.parts/.spark-staging-158 dst=null perm=null proto=rpc
17:37:52.334 INFO SparkHadoopWriter - Write Job job_202510271737517782725299037987949_0158 committed. Elapsed time: 8 ms.
17:37:52.334 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest1_5caa91ea-3dd8-4cc9-baa0-a6274b4a2a8b.bam.parts/header dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:37:52.335 WARN IntelDeflaterFactory - Intel Deflater not supported, using Java.util.zip.Deflater
17:37:52.336 INFO StateChange - BLOCK* allocate blk_1073741846_1022, replicas=127.0.0.1:37927 for /user/runner/ReadsSparkSinkUnitTest1_5caa91ea-3dd8-4cc9-baa0-a6274b4a2a8b.bam.parts/header
17:37:52.337 INFO DataNode - Receiving BP-617366194-10.1.0.252-1761586664431:blk_1073741846_1022 src: /127.0.0.1:57300 dest: /127.0.0.1:37927
17:37:52.338 INFO clienttrace - src: /127.0.0.1:57300, dest: /127.0.0.1:37927, bytes: 5901, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1369393814_1, offset: 0, srvID: 379dd5e5-59c3-42cd-b9d3-67ca5e44560b, blockid: BP-617366194-10.1.0.252-1761586664431:blk_1073741846_1022, duration(ns): 551057
17:37:52.338 INFO DataNode - PacketResponder: BP-617366194-10.1.0.252-1761586664431:blk_1073741846_1022, type=LAST_IN_PIPELINE terminating
17:37:52.339 INFO FSNamesystem - BLOCK* blk_1073741846_1022 is COMMITTED but not COMPLETE(numNodes= 0 < minimum = 1) in file /user/runner/ReadsSparkSinkUnitTest1_5caa91ea-3dd8-4cc9-baa0-a6274b4a2a8b.bam.parts/header
17:37:52.740 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest1_5caa91ea-3dd8-4cc9-baa0-a6274b4a2a8b.bam.parts/header is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:37:52.741 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest1_5caa91ea-3dd8-4cc9-baa0-a6274b4a2a8b.bam.parts/terminator dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:37:52.742 INFO StateChange - BLOCK* allocate blk_1073741847_1023, replicas=127.0.0.1:37927 for /user/runner/ReadsSparkSinkUnitTest1_5caa91ea-3dd8-4cc9-baa0-a6274b4a2a8b.bam.parts/terminator
17:37:52.743 INFO DataNode - Receiving BP-617366194-10.1.0.252-1761586664431:blk_1073741847_1023 src: /127.0.0.1:57306 dest: /127.0.0.1:37927
17:37:52.745 INFO clienttrace - src: /127.0.0.1:57306, dest: /127.0.0.1:37927, bytes: 28, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1369393814_1, offset: 0, srvID: 379dd5e5-59c3-42cd-b9d3-67ca5e44560b, blockid: BP-617366194-10.1.0.252-1761586664431:blk_1073741847_1023, duration(ns): 556897
17:37:52.745 INFO DataNode - PacketResponder: BP-617366194-10.1.0.252-1761586664431:blk_1073741847_1023, type=LAST_IN_PIPELINE terminating
17:37:52.745 INFO FSNamesystem - BLOCK* blk_1073741847_1023 is COMMITTED but not COMPLETE(numNodes= 0 < minimum = 1) in file /user/runner/ReadsSparkSinkUnitTest1_5caa91ea-3dd8-4cc9-baa0-a6274b4a2a8b.bam.parts/terminator
17:37:53.146 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest1_5caa91ea-3dd8-4cc9-baa0-a6274b4a2a8b.bam.parts/terminator is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:37:53.148 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=listStatus src=/user/runner/ReadsSparkSinkUnitTest1_5caa91ea-3dd8-4cc9-baa0-a6274b4a2a8b.bam.parts dst=null perm=null proto=rpc
17:37:53.149 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest1_5caa91ea-3dd8-4cc9-baa0-a6274b4a2a8b.bam.parts/output dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:37:53.150 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest1_5caa91ea-3dd8-4cc9-baa0-a6274b4a2a8b.bam.parts/output is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:37:53.150 INFO HadoopFileSystemWrapper - Concatenating 3 parts to hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest1_5caa91ea-3dd8-4cc9-baa0-a6274b4a2a8b.bam
17:37:53.151 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=concat src=[/user/runner/ReadsSparkSinkUnitTest1_5caa91ea-3dd8-4cc9-baa0-a6274b4a2a8b.bam.parts/header, /user/runner/ReadsSparkSinkUnitTest1_5caa91ea-3dd8-4cc9-baa0-a6274b4a2a8b.bam.parts/part-r-00000, /user/runner/ReadsSparkSinkUnitTest1_5caa91ea-3dd8-4cc9-baa0-a6274b4a2a8b.bam.parts/terminator] dst=/user/runner/ReadsSparkSinkUnitTest1_5caa91ea-3dd8-4cc9-baa0-a6274b4a2a8b.bam.parts/output perm=runner:supergroup:rw-r--r-- proto=rpc
17:37:53.152 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_5caa91ea-3dd8-4cc9-baa0-a6274b4a2a8b.bam dst=null perm=null proto=rpc
17:37:53.152 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=rename src=/user/runner/ReadsSparkSinkUnitTest1_5caa91ea-3dd8-4cc9-baa0-a6274b4a2a8b.bam.parts/output dst=/user/runner/ReadsSparkSinkUnitTest1_5caa91ea-3dd8-4cc9-baa0-a6274b4a2a8b.bam perm=runner:supergroup:rw-r--r-- proto=rpc
17:37:53.153 INFO HadoopFileSystemWrapper - Concatenating to hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest1_5caa91ea-3dd8-4cc9-baa0-a6274b4a2a8b.bam done
17:37:53.153 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_5caa91ea-3dd8-4cc9-baa0-a6274b4a2a8b.bam dst=null perm=null proto=rpc
17:37:53.153 INFO IndexFileMerger - Merging .sbi files in temp directory hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest1_5caa91ea-3dd8-4cc9-baa0-a6274b4a2a8b.bam.parts/ to hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest1_5caa91ea-3dd8-4cc9-baa0-a6274b4a2a8b.bam.sbi
17:37:53.154 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=listStatus src=/user/runner/ReadsSparkSinkUnitTest1_5caa91ea-3dd8-4cc9-baa0-a6274b4a2a8b.bam.parts dst=null perm=null proto=rpc
17:37:53.155 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest1_5caa91ea-3dd8-4cc9-baa0-a6274b4a2a8b.bam.sbi dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:37:53.156 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_5caa91ea-3dd8-4cc9-baa0-a6274b4a2a8b.bam.parts/.part-r-00000.sbi dst=null perm=null proto=rpc
17:37:53.157 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_5caa91ea-3dd8-4cc9-baa0-a6274b4a2a8b.bam.parts/.part-r-00000.sbi dst=null perm=null proto=rpc
17:37:53.158 WARN DFSUtil - Unexpected value for data transfer bytes=216 duration=0
17:37:53.159 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=delete src=/user/runner/ReadsSparkSinkUnitTest1_5caa91ea-3dd8-4cc9-baa0-a6274b4a2a8b.bam.parts/.part-r-00000.sbi dst=null perm=null proto=rpc
17:37:53.160 INFO StateChange - BLOCK* allocate blk_1073741848_1024, replicas=127.0.0.1:37927 for /user/runner/ReadsSparkSinkUnitTest1_5caa91ea-3dd8-4cc9-baa0-a6274b4a2a8b.bam.sbi
17:37:53.161 INFO DataNode - Receiving BP-617366194-10.1.0.252-1761586664431:blk_1073741848_1024 src: /127.0.0.1:57322 dest: /127.0.0.1:37927
17:37:53.162 INFO clienttrace - src: /127.0.0.1:57322, dest: /127.0.0.1:37927, bytes: 212, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1369393814_1, offset: 0, srvID: 379dd5e5-59c3-42cd-b9d3-67ca5e44560b, blockid: BP-617366194-10.1.0.252-1761586664431:blk_1073741848_1024, duration(ns): 549265
17:37:53.162 INFO DataNode - PacketResponder: BP-617366194-10.1.0.252-1761586664431:blk_1073741848_1024, type=LAST_IN_PIPELINE terminating
17:37:53.163 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest1_5caa91ea-3dd8-4cc9-baa0-a6274b4a2a8b.bam.sbi is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:37:53.163 INFO IndexFileMerger - Done merging .sbi files
17:37:53.164 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=delete src=/user/runner/ReadsSparkSinkUnitTest1_5caa91ea-3dd8-4cc9-baa0-a6274b4a2a8b.bam.parts dst=null perm=null proto=rpc
17:37:53.173 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_5caa91ea-3dd8-4cc9-baa0-a6274b4a2a8b.bam.sbi dst=null perm=null proto=rpc
17:37:53.174 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_5caa91ea-3dd8-4cc9-baa0-a6274b4a2a8b.bam.sbi dst=null perm=null proto=rpc
17:37:53.175 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_5caa91ea-3dd8-4cc9-baa0-a6274b4a2a8b.bam.sbi dst=null perm=null proto=rpc
17:37:53.176 WARN DFSUtil - Unexpected value for data transfer bytes=216 duration=0
17:37:53.176 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_5caa91ea-3dd8-4cc9-baa0-a6274b4a2a8b.bam dst=null perm=null proto=rpc
17:37:53.177 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_5caa91ea-3dd8-4cc9-baa0-a6274b4a2a8b.bam dst=null perm=null proto=rpc
17:37:53.178 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_5caa91ea-3dd8-4cc9-baa0-a6274b4a2a8b.bam dst=null perm=null proto=rpc
17:37:53.178 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_5caa91ea-3dd8-4cc9-baa0-a6274b4a2a8b.bam dst=null perm=null proto=rpc
17:37:53.179 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_5caa91ea-3dd8-4cc9-baa0-a6274b4a2a8b.bam.bai dst=null perm=null proto=rpc
17:37:53.180 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_5caa91ea-3dd8-4cc9-baa0-a6274b4a2a8b.bai dst=null perm=null proto=rpc
17:37:53.181 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:37:53.181 WARN DFSUtil - Unexpected value for data transfer bytes=5949 duration=0
17:37:53.183 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:53.183 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_5caa91ea-3dd8-4cc9-baa0-a6274b4a2a8b.bam.sbi dst=null perm=null proto=rpc
17:37:53.184 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_5caa91ea-3dd8-4cc9-baa0-a6274b4a2a8b.bam.sbi dst=null perm=null proto=rpc
17:37:53.184 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_5caa91ea-3dd8-4cc9-baa0-a6274b4a2a8b.bam.sbi dst=null perm=null proto=rpc
17:37:53.185 WARN DFSUtil - Unexpected value for data transfer bytes=216 duration=0
17:37:53.186 INFO MemoryStore - Block broadcast_72 stored as values in memory (estimated size 320.0 B, free 1918.0 MiB)
17:37:53.186 INFO MemoryStore - Block broadcast_72_piece0 stored as bytes in memory (estimated size 233.0 B, free 1918.0 MiB)
17:37:53.186 INFO BlockManagerInfo - Added broadcast_72_piece0 in memory on localhost:37953 (size: 233.0 B, free: 1919.7 MiB)
17:37:53.187 INFO SparkContext - Created broadcast 72 from broadcast at BamSource.java:104
17:37:53.188 INFO MemoryStore - Block broadcast_73 stored as values in memory (estimated size 297.9 KiB, free 1917.7 MiB)
17:37:53.194 INFO MemoryStore - Block broadcast_73_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1917.7 MiB)
17:37:53.194 INFO BlockManagerInfo - Added broadcast_73_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.6 MiB)
17:37:53.195 INFO SparkContext - Created broadcast 73 from newAPIHadoopFile at PathSplitSource.java:96
17:37:53.205 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_5caa91ea-3dd8-4cc9-baa0-a6274b4a2a8b.bam dst=null perm=null proto=rpc
17:37:53.205 INFO FileInputFormat - Total input files to process : 1
17:37:53.206 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_5caa91ea-3dd8-4cc9-baa0-a6274b4a2a8b.bam dst=null perm=null proto=rpc
17:37:53.221 INFO SparkContext - Starting job: collect at ReadsSparkSinkUnitTest.java:182
17:37:53.221 INFO DAGScheduler - Got job 33 (collect at ReadsSparkSinkUnitTest.java:182) with 1 output partitions
17:37:53.221 INFO DAGScheduler - Final stage: ResultStage 46 (collect at ReadsSparkSinkUnitTest.java:182)
17:37:53.221 INFO DAGScheduler - Parents of final stage: List()
17:37:53.221 INFO DAGScheduler - Missing parents: List()
17:37:53.222 INFO DAGScheduler - Submitting ResultStage 46 (MapPartitionsRDD[164] at filter at ReadsSparkSource.java:96), which has no missing parents
17:37:53.228 INFO MemoryStore - Block broadcast_74 stored as values in memory (estimated size 148.2 KiB, free 1917.5 MiB)
17:37:53.229 INFO MemoryStore - Block broadcast_74_piece0 stored as bytes in memory (estimated size 54.6 KiB, free 1917.5 MiB)
17:37:53.229 INFO BlockManagerInfo - Added broadcast_74_piece0 in memory on localhost:37953 (size: 54.6 KiB, free: 1919.6 MiB)
17:37:53.229 INFO SparkContext - Created broadcast 74 from broadcast at DAGScheduler.scala:1580
17:37:53.229 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 46 (MapPartitionsRDD[164] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:37:53.229 INFO TaskSchedulerImpl - Adding task set 46.0 with 1 tasks resource profile 0
17:37:53.230 INFO TaskSetManager - Starting task 0.0 in stage 46.0 (TID 84) (localhost, executor driver, partition 0, ANY, 7852 bytes)
17:37:53.231 INFO Executor - Running task 0.0 in stage 46.0 (TID 84)
17:37:53.244 INFO NewHadoopRDD - Input split: hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest1_5caa91ea-3dd8-4cc9-baa0-a6274b4a2a8b.bam:0+238969
17:37:53.246 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_5caa91ea-3dd8-4cc9-baa0-a6274b4a2a8b.bam dst=null perm=null proto=rpc
17:37:53.247 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_5caa91ea-3dd8-4cc9-baa0-a6274b4a2a8b.bam dst=null perm=null proto=rpc
17:37:53.248 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_5caa91ea-3dd8-4cc9-baa0-a6274b4a2a8b.bam.bai dst=null perm=null proto=rpc
17:37:53.249 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_5caa91ea-3dd8-4cc9-baa0-a6274b4a2a8b.bai dst=null perm=null proto=rpc
17:37:53.249 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:37:53.250 WARN DFSUtil - Unexpected value for data transfer bytes=5949 duration=0
17:37:53.253 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:53.254 WARN DFSUtil - Unexpected value for data transfer bytes=32 duration=0
17:37:53.257 INFO Executor - Finished task 0.0 in stage 46.0 (TID 84). 651526 bytes result sent to driver
17:37:53.259 INFO TaskSetManager - Finished task 0.0 in stage 46.0 (TID 84) in 29 ms on localhost (executor driver) (1/1)
17:37:53.259 INFO TaskSchedulerImpl - Removed TaskSet 46.0, whose tasks have all completed, from pool
17:37:53.259 INFO DAGScheduler - ResultStage 46 (collect at ReadsSparkSinkUnitTest.java:182) finished in 0.037 s
17:37:53.260 INFO DAGScheduler - Job 33 is finished. Cancelling potential speculative or zombie tasks for this job
17:37:53.260 INFO TaskSchedulerImpl - Killing all running tasks in stage 46: Stage finished
17:37:53.260 INFO DAGScheduler - Job 33 finished: collect at ReadsSparkSinkUnitTest.java:182, took 0.038999 s
17:37:53.271 INFO SparkContext - Starting job: count at ReadsSparkSinkUnitTest.java:185
17:37:53.271 INFO DAGScheduler - Got job 34 (count at ReadsSparkSinkUnitTest.java:185) with 1 output partitions
17:37:53.271 INFO DAGScheduler - Final stage: ResultStage 47 (count at ReadsSparkSinkUnitTest.java:185)
17:37:53.271 INFO DAGScheduler - Parents of final stage: List()
17:37:53.271 INFO DAGScheduler - Missing parents: List()
17:37:53.272 INFO DAGScheduler - Submitting ResultStage 47 (MapPartitionsRDD[146] at filter at ReadsSparkSource.java:96), which has no missing parents
17:37:53.287 INFO MemoryStore - Block broadcast_75 stored as values in memory (estimated size 426.1 KiB, free 1917.1 MiB)
17:37:53.293 INFO MemoryStore - Block broadcast_75_piece0 stored as bytes in memory (estimated size 153.6 KiB, free 1916.9 MiB)
17:37:53.294 INFO BlockManagerInfo - Added broadcast_75_piece0 in memory on localhost:37953 (size: 153.6 KiB, free: 1919.4 MiB)
17:37:53.294 INFO SparkContext - Created broadcast 75 from broadcast at DAGScheduler.scala:1580
17:37:53.294 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 47 (MapPartitionsRDD[146] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:37:53.294 INFO TaskSchedulerImpl - Adding task set 47.0 with 1 tasks resource profile 0
17:37:53.295 INFO TaskSetManager - Starting task 0.0 in stage 47.0 (TID 85) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7872 bytes)
17:37:53.295 INFO BlockManagerInfo - Removed broadcast_70_piece0 on localhost:37953 in memory (size: 166.1 KiB, free: 1919.6 MiB)
17:37:53.296 INFO Executor - Running task 0.0 in stage 47.0 (TID 85)
17:37:53.297 INFO BlockManagerInfo - Removed broadcast_69_piece0 on localhost:37953 in memory (size: 9.6 KiB, free: 1919.6 MiB)
17:37:53.299 INFO BlockManagerInfo - Removed broadcast_71_piece0 on localhost:37953 in memory (size: 67.1 KiB, free: 1919.6 MiB)
17:37:53.302 INFO BlockManagerInfo - Removed broadcast_74_piece0 on localhost:37953 in memory (size: 54.6 KiB, free: 1919.7 MiB)
17:37:53.303 INFO BlockManagerInfo - Removed broadcast_67_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.7 MiB)
17:37:53.303 INFO BlockManagerInfo - Removed broadcast_68_piece0 on localhost:37953 in memory (size: 9.6 KiB, free: 1919.8 MiB)
17:37:53.333 INFO NewHadoopRDD - Input split: file:/home/runner/work/gatk/gatk/src/test/resources/org/broadinstitute/hellbender/tools/BQSR/HiSeq.1mb.1RG.2k_lines.bam:0+222075
17:37:53.334 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:37:53.336 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:37:53.344 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:37:53.349 INFO Executor - Finished task 0.0 in stage 47.0 (TID 85). 989 bytes result sent to driver
17:37:53.349 INFO TaskSetManager - Finished task 0.0 in stage 47.0 (TID 85) in 54 ms on localhost (executor driver) (1/1)
17:37:53.350 INFO TaskSchedulerImpl - Removed TaskSet 47.0, whose tasks have all completed, from pool
17:37:53.350 INFO DAGScheduler - ResultStage 47 (count at ReadsSparkSinkUnitTest.java:185) finished in 0.078 s
17:37:53.350 INFO DAGScheduler - Job 34 is finished. Cancelling potential speculative or zombie tasks for this job
17:37:53.350 INFO TaskSchedulerImpl - Killing all running tasks in stage 47: Stage finished
17:37:53.350 INFO DAGScheduler - Job 34 finished: count at ReadsSparkSinkUnitTest.java:185, took 0.079473 s
17:37:53.354 INFO SparkContext - Starting job: count at ReadsSparkSinkUnitTest.java:185
17:37:53.355 INFO DAGScheduler - Got job 35 (count at ReadsSparkSinkUnitTest.java:185) with 1 output partitions
17:37:53.355 INFO DAGScheduler - Final stage: ResultStage 48 (count at ReadsSparkSinkUnitTest.java:185)
17:37:53.355 INFO DAGScheduler - Parents of final stage: List()
17:37:53.355 INFO DAGScheduler - Missing parents: List()
17:37:53.355 INFO DAGScheduler - Submitting ResultStage 48 (MapPartitionsRDD[164] at filter at ReadsSparkSource.java:96), which has no missing parents
17:37:53.362 INFO MemoryStore - Block broadcast_76 stored as values in memory (estimated size 148.1 KiB, free 1918.6 MiB)
17:37:53.363 INFO MemoryStore - Block broadcast_76_piece0 stored as bytes in memory (estimated size 54.6 KiB, free 1918.6 MiB)
17:37:53.363 INFO BlockManagerInfo - Added broadcast_76_piece0 in memory on localhost:37953 (size: 54.6 KiB, free: 1919.7 MiB)
17:37:53.364 INFO SparkContext - Created broadcast 76 from broadcast at DAGScheduler.scala:1580
17:37:53.364 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 48 (MapPartitionsRDD[164] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:37:53.364 INFO TaskSchedulerImpl - Adding task set 48.0 with 1 tasks resource profile 0
17:37:53.366 INFO TaskSetManager - Starting task 0.0 in stage 48.0 (TID 86) (localhost, executor driver, partition 0, ANY, 7852 bytes)
17:37:53.366 INFO Executor - Running task 0.0 in stage 48.0 (TID 86)
17:37:53.385 INFO NewHadoopRDD - Input split: hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest1_5caa91ea-3dd8-4cc9-baa0-a6274b4a2a8b.bam:0+238969
17:37:53.386 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_5caa91ea-3dd8-4cc9-baa0-a6274b4a2a8b.bam dst=null perm=null proto=rpc
17:37:53.388 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_5caa91ea-3dd8-4cc9-baa0-a6274b4a2a8b.bam dst=null perm=null proto=rpc
17:37:53.389 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_5caa91ea-3dd8-4cc9-baa0-a6274b4a2a8b.bam.bai dst=null perm=null proto=rpc
17:37:53.390 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_5caa91ea-3dd8-4cc9-baa0-a6274b4a2a8b.bai dst=null perm=null proto=rpc
17:37:53.392 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:37:53.393 WARN DFSUtil - Unexpected value for data transfer bytes=5949 duration=0
17:37:53.396 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:53.397 WARN DFSUtil - Unexpected value for data transfer bytes=32 duration=0
17:37:53.399 INFO Executor - Finished task 0.0 in stage 48.0 (TID 86). 989 bytes result sent to driver
17:37:53.400 INFO TaskSetManager - Finished task 0.0 in stage 48.0 (TID 86) in 35 ms on localhost (executor driver) (1/1)
17:37:53.400 INFO TaskSchedulerImpl - Removed TaskSet 48.0, whose tasks have all completed, from pool
17:37:53.400 INFO DAGScheduler - ResultStage 48 (count at ReadsSparkSinkUnitTest.java:185) finished in 0.044 s
17:37:53.401 INFO DAGScheduler - Job 35 is finished. Cancelling potential speculative or zombie tasks for this job
17:37:53.401 INFO TaskSchedulerImpl - Killing all running tasks in stage 48: Stage finished
17:37:53.401 INFO DAGScheduler - Job 35 finished: count at ReadsSparkSinkUnitTest.java:185, took 0.046505 s
17:37:53.403 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:37:53.407 INFO MemoryStore - Block broadcast_77 stored as values in memory (estimated size 297.9 KiB, free 1918.3 MiB)
17:37:53.413 INFO MemoryStore - Block broadcast_77_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1918.2 MiB)
17:37:53.414 INFO BlockManagerInfo - Added broadcast_77_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.6 MiB)
17:37:53.414 INFO SparkContext - Created broadcast 77 from newAPIHadoopFile at PathSplitSource.java:96
17:37:53.439 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:37:53.442 INFO MemoryStore - Block broadcast_78 stored as values in memory (estimated size 297.9 KiB, free 1917.9 MiB)
17:37:53.448 INFO MemoryStore - Block broadcast_78_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1917.9 MiB)
17:37:53.448 INFO BlockManagerInfo - Added broadcast_78_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.6 MiB)
17:37:53.449 INFO SparkContext - Created broadcast 78 from newAPIHadoopFile at PathSplitSource.java:96
17:37:53.473 INFO FileInputFormat - Total input files to process : 1
17:37:53.476 INFO MemoryStore - Block broadcast_79 stored as values in memory (estimated size 160.7 KiB, free 1917.7 MiB)
17:37:53.477 INFO MemoryStore - Block broadcast_79_piece0 stored as bytes in memory (estimated size 9.6 KiB, free 1917.7 MiB)
17:37:53.478 INFO BlockManagerInfo - Added broadcast_79_piece0 in memory on localhost:37953 (size: 9.6 KiB, free: 1919.6 MiB)
17:37:53.478 INFO SparkContext - Created broadcast 79 from broadcast at ReadsSparkSink.java:133
17:37:53.479 WARN HtsjdkReadsRddStorage - Unrecognized write option: DISABLE
17:37:53.479 WARN HtsjdkReadsRddStorage - Unrecognized write option: DISABLE
17:37:53.480 INFO MemoryStore - Block broadcast_80 stored as values in memory (estimated size 163.2 KiB, free 1917.5 MiB)
17:37:53.482 INFO MemoryStore - Block broadcast_80_piece0 stored as bytes in memory (estimated size 9.6 KiB, free 1917.5 MiB)
17:37:53.482 INFO BlockManagerInfo - Added broadcast_80_piece0 in memory on localhost:37953 (size: 9.6 KiB, free: 1919.6 MiB)
17:37:53.482 INFO SparkContext - Created broadcast 80 from broadcast at BamSink.java:76
17:37:53.486 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_4e67e57c-d41c-4284-b991-dd9c2514421b.bam.parts dst=null perm=null proto=rpc
17:37:53.487 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:37:53.487 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:37:53.487 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:37:53.489 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=mkdirs src=/user/runner/ReadsSparkSinkUnitTest1_4e67e57c-d41c-4284-b991-dd9c2514421b.bam.parts/_temporary/0 dst=null perm=runner:supergroup:rwxr-xr-x proto=rpc
17:37:53.495 INFO SparkContext - Starting job: runJob at SparkHadoopWriter.scala:83
17:37:53.496 INFO DAGScheduler - Registering RDD 178 (mapToPair at SparkUtils.java:161) as input to shuffle 11
17:37:53.497 INFO DAGScheduler - Got job 36 (runJob at SparkHadoopWriter.scala:83) with 1 output partitions
17:37:53.497 INFO DAGScheduler - Final stage: ResultStage 50 (runJob at SparkHadoopWriter.scala:83)
17:37:53.497 INFO DAGScheduler - Parents of final stage: List(ShuffleMapStage 49)
17:37:53.497 INFO DAGScheduler - Missing parents: List(ShuffleMapStage 49)
17:37:53.497 INFO DAGScheduler - Submitting ShuffleMapStage 49 (MapPartitionsRDD[178] at mapToPair at SparkUtils.java:161), which has no missing parents
17:37:53.515 INFO MemoryStore - Block broadcast_81 stored as values in memory (estimated size 520.4 KiB, free 1917.0 MiB)
17:37:53.516 INFO MemoryStore - Block broadcast_81_piece0 stored as bytes in memory (estimated size 166.1 KiB, free 1916.9 MiB)
17:37:53.517 INFO BlockManagerInfo - Added broadcast_81_piece0 in memory on localhost:37953 (size: 166.1 KiB, free: 1919.4 MiB)
17:37:53.517 INFO SparkContext - Created broadcast 81 from broadcast at DAGScheduler.scala:1580
17:37:53.517 INFO DAGScheduler - Submitting 1 missing tasks from ShuffleMapStage 49 (MapPartitionsRDD[178] at mapToPair at SparkUtils.java:161) (first 15 tasks are for partitions Vector(0))
17:37:53.517 INFO TaskSchedulerImpl - Adding task set 49.0 with 1 tasks resource profile 0
17:37:53.518 INFO TaskSetManager - Starting task 0.0 in stage 49.0 (TID 87) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7861 bytes)
17:37:53.519 INFO Executor - Running task 0.0 in stage 49.0 (TID 87)
17:37:53.563 INFO NewHadoopRDD - Input split: file:/home/runner/work/gatk/gatk/src/test/resources/org/broadinstitute/hellbender/tools/BQSR/HiSeq.1mb.1RG.2k_lines.bam:0+222075
17:37:53.565 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:37:53.566 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:37:53.574 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:37:53.586 INFO Executor - Finished task 0.0 in stage 49.0 (TID 87). 1148 bytes result sent to driver
17:37:53.587 INFO TaskSetManager - Finished task 0.0 in stage 49.0 (TID 87) in 69 ms on localhost (executor driver) (1/1)
17:37:53.587 INFO TaskSchedulerImpl - Removed TaskSet 49.0, whose tasks have all completed, from pool
17:37:53.587 INFO DAGScheduler - ShuffleMapStage 49 (mapToPair at SparkUtils.java:161) finished in 0.089 s
17:37:53.587 INFO DAGScheduler - looking for newly runnable stages
17:37:53.587 INFO DAGScheduler - running: HashSet()
17:37:53.587 INFO DAGScheduler - waiting: HashSet(ResultStage 50)
17:37:53.588 INFO DAGScheduler - failed: HashSet()
17:37:53.588 INFO DAGScheduler - Submitting ResultStage 50 (MapPartitionsRDD[183] at mapToPair at BamSink.java:91), which has no missing parents
17:37:53.595 INFO MemoryStore - Block broadcast_82 stored as values in memory (estimated size 241.5 KiB, free 1916.6 MiB)
17:37:53.596 INFO MemoryStore - Block broadcast_82_piece0 stored as bytes in memory (estimated size 67.1 KiB, free 1916.6 MiB)
17:37:53.596 INFO BlockManagerInfo - Added broadcast_82_piece0 in memory on localhost:37953 (size: 67.1 KiB, free: 1919.4 MiB)
17:37:53.597 INFO SparkContext - Created broadcast 82 from broadcast at DAGScheduler.scala:1580
17:37:53.597 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 50 (MapPartitionsRDD[183] at mapToPair at BamSink.java:91) (first 15 tasks are for partitions Vector(0))
17:37:53.597 INFO TaskSchedulerImpl - Adding task set 50.0 with 1 tasks resource profile 0
17:37:53.598 INFO TaskSetManager - Starting task 0.0 in stage 50.0 (TID 88) (localhost, executor driver, partition 0, NODE_LOCAL, 7513 bytes)
17:37:53.599 INFO Executor - Running task 0.0 in stage 50.0 (TID 88)
17:37:53.607 INFO ShuffleBlockFetcherIterator - Getting 1 (343.8 KiB) non-empty blocks including 1 (343.8 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
17:37:53.607 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
17:37:53.622 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:37:53.622 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:37:53.622 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:37:53.622 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:37:53.622 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:37:53.622 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:37:53.624 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest1_4e67e57c-d41c-4284-b991-dd9c2514421b.bam.parts/_temporary/0/_temporary/attempt_202510271737537149224829568670059_0183_r_000000_0/part-r-00000 dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:37:53.624 WARN IntelDeflaterFactory - Intel Deflater not supported, using Java.util.zip.Deflater
17:37:53.629 INFO StateChange - BLOCK* allocate blk_1073741849_1025, replicas=127.0.0.1:37927 for /user/runner/ReadsSparkSinkUnitTest1_4e67e57c-d41c-4284-b991-dd9c2514421b.bam.parts/_temporary/0/_temporary/attempt_202510271737537149224829568670059_0183_r_000000_0/part-r-00000
17:37:53.631 INFO DataNode - Receiving BP-617366194-10.1.0.252-1761586664431:blk_1073741849_1025 src: /127.0.0.1:57324 dest: /127.0.0.1:37927
17:37:53.635 INFO clienttrace - src: /127.0.0.1:57324, dest: /127.0.0.1:37927, bytes: 233040, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1369393814_1, offset: 0, srvID: 379dd5e5-59c3-42cd-b9d3-67ca5e44560b, blockid: BP-617366194-10.1.0.252-1761586664431:blk_1073741849_1025, duration(ns): 3401339
17:37:53.635 INFO DataNode - PacketResponder: BP-617366194-10.1.0.252-1761586664431:blk_1073741849_1025, type=LAST_IN_PIPELINE terminating
17:37:53.636 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest1_4e67e57c-d41c-4284-b991-dd9c2514421b.bam.parts/_temporary/0/_temporary/attempt_202510271737537149224829568670059_0183_r_000000_0/part-r-00000 is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:37:53.637 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_4e67e57c-d41c-4284-b991-dd9c2514421b.bam.parts/_temporary/0/_temporary/attempt_202510271737537149224829568670059_0183_r_000000_0/part-r-00000 dst=null perm=null proto=rpc
17:37:53.638 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_4e67e57c-d41c-4284-b991-dd9c2514421b.bam.parts/_temporary/0/_temporary/attempt_202510271737537149224829568670059_0183_r_000000_0 dst=null perm=null proto=rpc
17:37:53.638 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_4e67e57c-d41c-4284-b991-dd9c2514421b.bam.parts/_temporary/0/_temporary/attempt_202510271737537149224829568670059_0183_r_000000_0 dst=null perm=null proto=rpc
17:37:53.639 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_4e67e57c-d41c-4284-b991-dd9c2514421b.bam.parts/_temporary/0/task_202510271737537149224829568670059_0183_r_000000 dst=null perm=null proto=rpc
17:37:53.640 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=rename src=/user/runner/ReadsSparkSinkUnitTest1_4e67e57c-d41c-4284-b991-dd9c2514421b.bam.parts/_temporary/0/_temporary/attempt_202510271737537149224829568670059_0183_r_000000_0 dst=/user/runner/ReadsSparkSinkUnitTest1_4e67e57c-d41c-4284-b991-dd9c2514421b.bam.parts/_temporary/0/task_202510271737537149224829568670059_0183_r_000000 perm=runner:supergroup:rwxr-xr-x proto=rpc
17:37:53.640 INFO FileOutputCommitter - Saved output of task 'attempt_202510271737537149224829568670059_0183_r_000000_0' to hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest1_4e67e57c-d41c-4284-b991-dd9c2514421b.bam.parts/_temporary/0/task_202510271737537149224829568670059_0183_r_000000
17:37:53.640 INFO SparkHadoopMapRedUtil - attempt_202510271737537149224829568670059_0183_r_000000_0: Committed. Elapsed time: 2 ms.
17:37:53.647 INFO Executor - Finished task 0.0 in stage 50.0 (TID 88). 1944 bytes result sent to driver
17:37:53.647 INFO BlockManagerInfo - Removed broadcast_73_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.4 MiB)
17:37:53.647 INFO TaskSetManager - Finished task 0.0 in stage 50.0 (TID 88) in 49 ms on localhost (executor driver) (1/1)
17:37:53.647 INFO TaskSchedulerImpl - Removed TaskSet 50.0, whose tasks have all completed, from pool
17:37:53.648 INFO DAGScheduler - ResultStage 50 (runJob at SparkHadoopWriter.scala:83) finished in 0.060 s
17:37:53.648 INFO DAGScheduler - Job 36 is finished. Cancelling potential speculative or zombie tasks for this job
17:37:53.648 INFO TaskSchedulerImpl - Killing all running tasks in stage 50: Stage finished
17:37:53.648 INFO DAGScheduler - Job 36 finished: runJob at SparkHadoopWriter.scala:83, took 0.152363 s
17:37:53.648 INFO BlockManagerInfo - Removed broadcast_75_piece0 on localhost:37953 in memory (size: 153.6 KiB, free: 1919.6 MiB)
17:37:53.649 INFO SparkHadoopWriter - Start to commit write Job job_202510271737537149224829568670059_0183.
17:37:53.650 INFO BlockManagerInfo - Removed broadcast_78_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.6 MiB)
17:37:53.650 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=listStatus src=/user/runner/ReadsSparkSinkUnitTest1_4e67e57c-d41c-4284-b991-dd9c2514421b.bam.parts/_temporary/0 dst=null perm=null proto=rpc
17:37:53.650 INFO BlockManagerInfo - Removed broadcast_72_piece0 on localhost:37953 in memory (size: 233.0 B, free: 1919.6 MiB)
17:37:53.651 INFO BlockManagerInfo - Removed broadcast_81_piece0 on localhost:37953 in memory (size: 166.1 KiB, free: 1919.8 MiB)
17:37:53.651 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_4e67e57c-d41c-4284-b991-dd9c2514421b.bam.parts dst=null perm=null proto=rpc
17:37:53.652 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=listStatus src=/user/runner/ReadsSparkSinkUnitTest1_4e67e57c-d41c-4284-b991-dd9c2514421b.bam.parts/_temporary/0/task_202510271737537149224829568670059_0183_r_000000 dst=null perm=null proto=rpc
17:37:53.652 INFO BlockManagerInfo - Removed broadcast_76_piece0 on localhost:37953 in memory (size: 54.6 KiB, free: 1919.8 MiB)
17:37:53.653 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_4e67e57c-d41c-4284-b991-dd9c2514421b.bam.parts/part-r-00000 dst=null perm=null proto=rpc
17:37:53.653 INFO BlockManagerInfo - Removed broadcast_66_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.9 MiB)
17:37:53.654 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=rename src=/user/runner/ReadsSparkSinkUnitTest1_4e67e57c-d41c-4284-b991-dd9c2514421b.bam.parts/_temporary/0/task_202510271737537149224829568670059_0183_r_000000/part-r-00000 dst=/user/runner/ReadsSparkSinkUnitTest1_4e67e57c-d41c-4284-b991-dd9c2514421b.bam.parts/part-r-00000 perm=runner:supergroup:rw-r--r-- proto=rpc
17:37:53.655 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=delete src=/user/runner/ReadsSparkSinkUnitTest1_4e67e57c-d41c-4284-b991-dd9c2514421b.bam.parts/_temporary dst=null perm=null proto=rpc
17:37:53.656 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest1_4e67e57c-d41c-4284-b991-dd9c2514421b.bam.parts/_SUCCESS dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:37:53.656 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest1_4e67e57c-d41c-4284-b991-dd9c2514421b.bam.parts/_SUCCESS is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:37:53.660 INFO audit - allowed=false ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=delete src=/user/runner/ReadsSparkSinkUnitTest1_4e67e57c-d41c-4284-b991-dd9c2514421b.bam.parts/.spark-staging-183 dst=null perm=null proto=rpc
17:37:53.660 INFO SparkHadoopWriter - Write Job job_202510271737537149224829568670059_0183 committed. Elapsed time: 11 ms.
17:37:53.663 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest1_4e67e57c-d41c-4284-b991-dd9c2514421b.bam.parts/header dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:37:53.663 WARN IntelDeflaterFactory - Intel Deflater not supported, using Java.util.zip.Deflater
17:37:53.665 INFO StateChange - BLOCK* allocate blk_1073741850_1026, replicas=127.0.0.1:37927 for /user/runner/ReadsSparkSinkUnitTest1_4e67e57c-d41c-4284-b991-dd9c2514421b.bam.parts/header
17:37:53.666 INFO DataNode - Receiving BP-617366194-10.1.0.252-1761586664431:blk_1073741850_1026 src: /127.0.0.1:57334 dest: /127.0.0.1:37927
17:37:53.667 INFO clienttrace - src: /127.0.0.1:57334, dest: /127.0.0.1:37927, bytes: 5901, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1369393814_1, offset: 0, srvID: 379dd5e5-59c3-42cd-b9d3-67ca5e44560b, blockid: BP-617366194-10.1.0.252-1761586664431:blk_1073741850_1026, duration(ns): 510829
17:37:53.667 INFO DataNode - PacketResponder: BP-617366194-10.1.0.252-1761586664431:blk_1073741850_1026, type=LAST_IN_PIPELINE terminating
17:37:53.668 INFO FSNamesystem - BLOCK* blk_1073741850_1026 is COMMITTED but not COMPLETE(numNodes= 0 < minimum = 1) in file /user/runner/ReadsSparkSinkUnitTest1_4e67e57c-d41c-4284-b991-dd9c2514421b.bam.parts/header
17:37:54.069 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest1_4e67e57c-d41c-4284-b991-dd9c2514421b.bam.parts/header is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:37:54.071 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest1_4e67e57c-d41c-4284-b991-dd9c2514421b.bam.parts/terminator dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:37:54.072 INFO StateChange - BLOCK* allocate blk_1073741851_1027, replicas=127.0.0.1:37927 for /user/runner/ReadsSparkSinkUnitTest1_4e67e57c-d41c-4284-b991-dd9c2514421b.bam.parts/terminator
17:37:54.073 INFO DataNode - Receiving BP-617366194-10.1.0.252-1761586664431:blk_1073741851_1027 src: /127.0.0.1:57350 dest: /127.0.0.1:37927
17:37:54.074 INFO clienttrace - src: /127.0.0.1:57350, dest: /127.0.0.1:37927, bytes: 28, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1369393814_1, offset: 0, srvID: 379dd5e5-59c3-42cd-b9d3-67ca5e44560b, blockid: BP-617366194-10.1.0.252-1761586664431:blk_1073741851_1027, duration(ns): 563158
17:37:54.075 INFO DataNode - PacketResponder: BP-617366194-10.1.0.252-1761586664431:blk_1073741851_1027, type=LAST_IN_PIPELINE terminating
17:37:54.075 INFO FSNamesystem - BLOCK* blk_1073741851_1027 is COMMITTED but not COMPLETE(numNodes= 0 < minimum = 1) in file /user/runner/ReadsSparkSinkUnitTest1_4e67e57c-d41c-4284-b991-dd9c2514421b.bam.parts/terminator
17:37:54.476 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest1_4e67e57c-d41c-4284-b991-dd9c2514421b.bam.parts/terminator is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:37:54.478 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=listStatus src=/user/runner/ReadsSparkSinkUnitTest1_4e67e57c-d41c-4284-b991-dd9c2514421b.bam.parts dst=null perm=null proto=rpc
17:37:54.479 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest1_4e67e57c-d41c-4284-b991-dd9c2514421b.bam.parts/output dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:37:54.480 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest1_4e67e57c-d41c-4284-b991-dd9c2514421b.bam.parts/output is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:37:54.480 INFO HadoopFileSystemWrapper - Concatenating 3 parts to hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest1_4e67e57c-d41c-4284-b991-dd9c2514421b.bam
17:37:54.481 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=concat src=[/user/runner/ReadsSparkSinkUnitTest1_4e67e57c-d41c-4284-b991-dd9c2514421b.bam.parts/header, /user/runner/ReadsSparkSinkUnitTest1_4e67e57c-d41c-4284-b991-dd9c2514421b.bam.parts/part-r-00000, /user/runner/ReadsSparkSinkUnitTest1_4e67e57c-d41c-4284-b991-dd9c2514421b.bam.parts/terminator] dst=/user/runner/ReadsSparkSinkUnitTest1_4e67e57c-d41c-4284-b991-dd9c2514421b.bam.parts/output perm=runner:supergroup:rw-r--r-- proto=rpc
17:37:54.481 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_4e67e57c-d41c-4284-b991-dd9c2514421b.bam dst=null perm=null proto=rpc
17:37:54.482 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=rename src=/user/runner/ReadsSparkSinkUnitTest1_4e67e57c-d41c-4284-b991-dd9c2514421b.bam.parts/output dst=/user/runner/ReadsSparkSinkUnitTest1_4e67e57c-d41c-4284-b991-dd9c2514421b.bam perm=runner:supergroup:rw-r--r-- proto=rpc
17:37:54.482 INFO HadoopFileSystemWrapper - Concatenating to hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest1_4e67e57c-d41c-4284-b991-dd9c2514421b.bam done
17:37:54.483 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_4e67e57c-d41c-4284-b991-dd9c2514421b.bam dst=null perm=null proto=rpc
17:37:54.484 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=delete src=/user/runner/ReadsSparkSinkUnitTest1_4e67e57c-d41c-4284-b991-dd9c2514421b.bam.parts dst=null perm=null proto=rpc
17:37:54.484 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_4e67e57c-d41c-4284-b991-dd9c2514421b.bam dst=null perm=null proto=rpc
17:37:54.485 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_4e67e57c-d41c-4284-b991-dd9c2514421b.bam dst=null perm=null proto=rpc
17:37:54.485 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_4e67e57c-d41c-4284-b991-dd9c2514421b.bam dst=null perm=null proto=rpc
17:37:54.486 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_4e67e57c-d41c-4284-b991-dd9c2514421b.bam dst=null perm=null proto=rpc
17:37:54.487 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_4e67e57c-d41c-4284-b991-dd9c2514421b.bam.bai dst=null perm=null proto=rpc
17:37:54.487 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_4e67e57c-d41c-4284-b991-dd9c2514421b.bai dst=null perm=null proto=rpc
17:37:54.488 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:37:54.489 WARN DFSUtil - Unexpected value for data transfer bytes=5949 duration=0
17:37:54.490 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:54.491 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_4e67e57c-d41c-4284-b991-dd9c2514421b.bam.sbi dst=null perm=null proto=rpc
17:37:54.493 INFO MemoryStore - Block broadcast_83 stored as values in memory (estimated size 297.9 KiB, free 1918.7 MiB)
17:37:54.498 INFO MemoryStore - Block broadcast_83_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1918.7 MiB)
17:37:54.499 INFO BlockManagerInfo - Added broadcast_83_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.8 MiB)
17:37:54.499 INFO SparkContext - Created broadcast 83 from newAPIHadoopFile at PathSplitSource.java:96
17:37:54.519 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_4e67e57c-d41c-4284-b991-dd9c2514421b.bam dst=null perm=null proto=rpc
17:37:54.520 INFO FileInputFormat - Total input files to process : 1
17:37:54.520 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_4e67e57c-d41c-4284-b991-dd9c2514421b.bam dst=null perm=null proto=rpc
17:37:54.553 INFO SparkContext - Starting job: collect at ReadsSparkSinkUnitTest.java:182
17:37:54.554 INFO DAGScheduler - Got job 37 (collect at ReadsSparkSinkUnitTest.java:182) with 1 output partitions
17:37:54.554 INFO DAGScheduler - Final stage: ResultStage 51 (collect at ReadsSparkSinkUnitTest.java:182)
17:37:54.554 INFO DAGScheduler - Parents of final stage: List()
17:37:54.554 INFO DAGScheduler - Missing parents: List()
17:37:54.554 INFO DAGScheduler - Submitting ResultStage 51 (MapPartitionsRDD[190] at filter at ReadsSparkSource.java:96), which has no missing parents
17:37:54.570 INFO MemoryStore - Block broadcast_84 stored as values in memory (estimated size 426.2 KiB, free 1918.3 MiB)
17:37:54.571 INFO MemoryStore - Block broadcast_84_piece0 stored as bytes in memory (estimated size 153.7 KiB, free 1918.1 MiB)
17:37:54.572 INFO BlockManagerInfo - Added broadcast_84_piece0 in memory on localhost:37953 (size: 153.7 KiB, free: 1919.7 MiB)
17:37:54.572 INFO SparkContext - Created broadcast 84 from broadcast at DAGScheduler.scala:1580
17:37:54.572 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 51 (MapPartitionsRDD[190] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:37:54.572 INFO TaskSchedulerImpl - Adding task set 51.0 with 1 tasks resource profile 0
17:37:54.573 INFO TaskSetManager - Starting task 0.0 in stage 51.0 (TID 89) (localhost, executor driver, partition 0, ANY, 7852 bytes)
17:37:54.573 INFO Executor - Running task 0.0 in stage 51.0 (TID 89)
17:37:54.598 INFO FsDatasetAsyncDiskService - Scheduling blk_1073741840_1016 replica FinalizedReplica, blk_1073741840_1016, FINALIZED
getNumBytes() = 5472
getBytesOnDisk() = 5472
getVisibleLength()= 5472
getVolume() = /tmp/minicluster_storage14473397491642437130/data/data2
getBlockURI() = file:/tmp/minicluster_storage14473397491642437130/data/data2/current/BP-617366194-10.1.0.252-1761586664431/current/finalized/subdir0/subdir0/blk_1073741840 for deletion
17:37:54.598 INFO FsDatasetAsyncDiskService - Scheduling blk_1073741845_1021 replica FinalizedReplica, blk_1073741845_1021, FINALIZED
getNumBytes() = 212
getBytesOnDisk() = 212
getVisibleLength()= 212
getVolume() = /tmp/minicluster_storage14473397491642437130/data/data1
getBlockURI() = file:/tmp/minicluster_storage14473397491642437130/data/data1/current/BP-617366194-10.1.0.252-1761586664431/current/finalized/subdir0/subdir0/blk_1073741845 for deletion
17:37:54.598 INFO FsDatasetAsyncDiskService - Deleted BP-617366194-10.1.0.252-1761586664431 blk_1073741840_1016 URI file:/tmp/minicluster_storage14473397491642437130/data/data2/current/BP-617366194-10.1.0.252-1761586664431/current/finalized/subdir0/subdir0/blk_1073741840
17:37:54.598 INFO FsDatasetAsyncDiskService - Deleted BP-617366194-10.1.0.252-1761586664431 blk_1073741845_1021 URI file:/tmp/minicluster_storage14473397491642437130/data/data1/current/BP-617366194-10.1.0.252-1761586664431/current/finalized/subdir0/subdir0/blk_1073741845
17:37:54.607 INFO NewHadoopRDD - Input split: hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest1_4e67e57c-d41c-4284-b991-dd9c2514421b.bam:0+238969
17:37:54.608 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_4e67e57c-d41c-4284-b991-dd9c2514421b.bam dst=null perm=null proto=rpc
17:37:54.609 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_4e67e57c-d41c-4284-b991-dd9c2514421b.bam dst=null perm=null proto=rpc
17:37:54.611 WARN DFSUtil - Unexpected value for data transfer bytes=5949 duration=0
17:37:54.611 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_4e67e57c-d41c-4284-b991-dd9c2514421b.bam dst=null perm=null proto=rpc
17:37:54.612 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_4e67e57c-d41c-4284-b991-dd9c2514421b.bam dst=null perm=null proto=rpc
17:37:54.613 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_4e67e57c-d41c-4284-b991-dd9c2514421b.bam.bai dst=null perm=null proto=rpc
17:37:54.613 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_4e67e57c-d41c-4284-b991-dd9c2514421b.bai dst=null perm=null proto=rpc
17:37:54.614 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:37:54.615 WARN DFSUtil - Unexpected value for data transfer bytes=5949 duration=0
17:37:54.616 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_4e67e57c-d41c-4284-b991-dd9c2514421b.bam dst=null perm=null proto=rpc
17:37:54.617 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_4e67e57c-d41c-4284-b991-dd9c2514421b.bam dst=null perm=null proto=rpc
17:37:54.618 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:37:54.618 WARN DFSUtil - Unexpected value for data transfer bytes=5949 duration=0
17:37:54.623 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:54.624 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:54.625 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:54.627 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:54.628 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:54.629 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:54.630 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:54.632 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:54.633 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:54.634 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:54.635 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:54.636 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:54.636 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:54.637 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:54.638 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:54.640 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:54.641 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:54.642 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:54.643 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:54.644 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:54.644 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:54.645 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:54.646 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:54.647 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:54.648 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:54.649 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:54.650 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:54.651 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:54.652 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:54.653 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:54.655 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:54.657 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:54.658 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:54.658 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:54.659 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:54.660 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:54.661 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:54.662 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:54.663 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:54.664 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:54.665 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:54.666 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:54.667 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:54.668 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:54.669 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:54.670 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:54.671 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:54.672 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:54.672 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:54.673 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:54.674 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:54.674 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:54.676 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:54.677 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:54.678 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:54.679 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_4e67e57c-d41c-4284-b991-dd9c2514421b.bam dst=null perm=null proto=rpc
17:37:54.680 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_4e67e57c-d41c-4284-b991-dd9c2514421b.bam dst=null perm=null proto=rpc
17:37:54.681 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_4e67e57c-d41c-4284-b991-dd9c2514421b.bam.bai dst=null perm=null proto=rpc
17:37:54.681 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_4e67e57c-d41c-4284-b991-dd9c2514421b.bai dst=null perm=null proto=rpc
17:37:54.682 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:37:54.683 WARN DFSUtil - Unexpected value for data transfer bytes=5949 duration=0
17:37:54.686 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:54.689 INFO Executor - Finished task 0.0 in stage 51.0 (TID 89). 651483 bytes result sent to driver
17:37:54.691 INFO TaskSetManager - Finished task 0.0 in stage 51.0 (TID 89) in 118 ms on localhost (executor driver) (1/1)
17:37:54.691 INFO TaskSchedulerImpl - Removed TaskSet 51.0, whose tasks have all completed, from pool
17:37:54.691 INFO DAGScheduler - ResultStage 51 (collect at ReadsSparkSinkUnitTest.java:182) finished in 0.136 s
17:37:54.691 INFO DAGScheduler - Job 37 is finished. Cancelling potential speculative or zombie tasks for this job
17:37:54.691 INFO TaskSchedulerImpl - Killing all running tasks in stage 51: Stage finished
17:37:54.691 INFO DAGScheduler - Job 37 finished: collect at ReadsSparkSinkUnitTest.java:182, took 0.137920 s
17:37:54.702 INFO SparkContext - Starting job: count at ReadsSparkSinkUnitTest.java:185
17:37:54.703 INFO DAGScheduler - Got job 38 (count at ReadsSparkSinkUnitTest.java:185) with 1 output partitions
17:37:54.703 INFO DAGScheduler - Final stage: ResultStage 52 (count at ReadsSparkSinkUnitTest.java:185)
17:37:54.703 INFO DAGScheduler - Parents of final stage: List()
17:37:54.703 INFO DAGScheduler - Missing parents: List()
17:37:54.703 INFO DAGScheduler - Submitting ResultStage 52 (MapPartitionsRDD[171] at filter at ReadsSparkSource.java:96), which has no missing parents
17:37:54.721 INFO MemoryStore - Block broadcast_85 stored as values in memory (estimated size 426.1 KiB, free 1917.7 MiB)
17:37:54.722 INFO MemoryStore - Block broadcast_85_piece0 stored as bytes in memory (estimated size 153.6 KiB, free 1917.6 MiB)
17:37:54.723 INFO BlockManagerInfo - Added broadcast_85_piece0 in memory on localhost:37953 (size: 153.6 KiB, free: 1919.5 MiB)
17:37:54.723 INFO SparkContext - Created broadcast 85 from broadcast at DAGScheduler.scala:1580
17:37:54.723 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 52 (MapPartitionsRDD[171] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:37:54.723 INFO TaskSchedulerImpl - Adding task set 52.0 with 1 tasks resource profile 0
17:37:54.724 INFO TaskSetManager - Starting task 0.0 in stage 52.0 (TID 90) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7872 bytes)
17:37:54.725 INFO Executor - Running task 0.0 in stage 52.0 (TID 90)
17:37:54.759 INFO NewHadoopRDD - Input split: file:/home/runner/work/gatk/gatk/src/test/resources/org/broadinstitute/hellbender/tools/BQSR/HiSeq.1mb.1RG.2k_lines.bam:0+222075
17:37:54.760 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:37:54.761 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:37:54.768 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:37:54.772 INFO Executor - Finished task 0.0 in stage 52.0 (TID 90). 989 bytes result sent to driver
17:37:54.773 INFO TaskSetManager - Finished task 0.0 in stage 52.0 (TID 90) in 49 ms on localhost (executor driver) (1/1)
17:37:54.773 INFO TaskSchedulerImpl - Removed TaskSet 52.0, whose tasks have all completed, from pool
17:37:54.773 INFO DAGScheduler - ResultStage 52 (count at ReadsSparkSinkUnitTest.java:185) finished in 0.070 s
17:37:54.773 INFO DAGScheduler - Job 38 is finished. Cancelling potential speculative or zombie tasks for this job
17:37:54.773 INFO TaskSchedulerImpl - Killing all running tasks in stage 52: Stage finished
17:37:54.773 INFO DAGScheduler - Job 38 finished: count at ReadsSparkSinkUnitTest.java:185, took 0.071111 s
17:37:54.777 INFO SparkContext - Starting job: count at ReadsSparkSinkUnitTest.java:185
17:37:54.777 INFO DAGScheduler - Got job 39 (count at ReadsSparkSinkUnitTest.java:185) with 1 output partitions
17:37:54.777 INFO DAGScheduler - Final stage: ResultStage 53 (count at ReadsSparkSinkUnitTest.java:185)
17:37:54.777 INFO DAGScheduler - Parents of final stage: List()
17:37:54.777 INFO DAGScheduler - Missing parents: List()
17:37:54.777 INFO DAGScheduler - Submitting ResultStage 53 (MapPartitionsRDD[190] at filter at ReadsSparkSource.java:96), which has no missing parents
17:37:54.796 INFO MemoryStore - Block broadcast_86 stored as values in memory (estimated size 426.1 KiB, free 1917.1 MiB)
17:37:54.797 INFO MemoryStore - Block broadcast_86_piece0 stored as bytes in memory (estimated size 153.6 KiB, free 1917.0 MiB)
17:37:54.798 INFO BlockManagerInfo - Added broadcast_86_piece0 in memory on localhost:37953 (size: 153.6 KiB, free: 1919.4 MiB)
17:37:54.798 INFO SparkContext - Created broadcast 86 from broadcast at DAGScheduler.scala:1580
17:37:54.799 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 53 (MapPartitionsRDD[190] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:37:54.799 INFO TaskSchedulerImpl - Adding task set 53.0 with 1 tasks resource profile 0
17:37:54.800 INFO TaskSetManager - Starting task 0.0 in stage 53.0 (TID 91) (localhost, executor driver, partition 0, ANY, 7852 bytes)
17:37:54.801 INFO Executor - Running task 0.0 in stage 53.0 (TID 91)
17:37:54.835 INFO NewHadoopRDD - Input split: hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest1_4e67e57c-d41c-4284-b991-dd9c2514421b.bam:0+238969
17:37:54.836 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_4e67e57c-d41c-4284-b991-dd9c2514421b.bam dst=null perm=null proto=rpc
17:37:54.837 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_4e67e57c-d41c-4284-b991-dd9c2514421b.bam dst=null perm=null proto=rpc
17:37:54.838 WARN DFSUtil - Unexpected value for data transfer bytes=5949 duration=0
17:37:54.838 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_4e67e57c-d41c-4284-b991-dd9c2514421b.bam dst=null perm=null proto=rpc
17:37:54.839 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_4e67e57c-d41c-4284-b991-dd9c2514421b.bam dst=null perm=null proto=rpc
17:37:54.840 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_4e67e57c-d41c-4284-b991-dd9c2514421b.bam.bai dst=null perm=null proto=rpc
17:37:54.841 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_4e67e57c-d41c-4284-b991-dd9c2514421b.bai dst=null perm=null proto=rpc
17:37:54.842 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:37:54.843 WARN DFSUtil - Unexpected value for data transfer bytes=5949 duration=0
17:37:54.845 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_4e67e57c-d41c-4284-b991-dd9c2514421b.bam dst=null perm=null proto=rpc
17:37:54.846 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_4e67e57c-d41c-4284-b991-dd9c2514421b.bam dst=null perm=null proto=rpc
17:37:54.846 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:37:54.847 WARN DFSUtil - Unexpected value for data transfer bytes=5949 duration=0
17:37:54.851 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:54.852 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:54.853 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:54.854 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:54.855 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:54.856 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:54.857 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:54.858 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:54.859 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:54.861 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:54.861 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:54.862 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:54.863 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:54.864 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:54.866 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:54.866 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:54.867 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:54.872 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:54.873 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:54.873 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:54.874 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:54.875 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:54.876 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:54.877 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:54.878 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:54.880 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:54.880 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:54.881 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:54.882 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:54.883 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:54.885 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:54.886 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:54.887 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:54.888 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:54.888 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:54.889 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:54.890 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:54.891 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:54.892 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:54.893 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:54.894 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:54.895 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:54.895 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:54.896 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:54.897 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:54.898 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:54.899 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:54.900 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:54.901 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:54.902 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:54.903 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:54.904 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:54.905 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:54.906 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:54.907 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:54.907 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:54.908 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:54.909 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:54.909 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:54.910 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:37:54.911 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:54.913 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:54.914 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_4e67e57c-d41c-4284-b991-dd9c2514421b.bam dst=null perm=null proto=rpc
17:37:54.914 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_4e67e57c-d41c-4284-b991-dd9c2514421b.bam dst=null perm=null proto=rpc
17:37:54.915 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_4e67e57c-d41c-4284-b991-dd9c2514421b.bam.bai dst=null perm=null proto=rpc
17:37:54.916 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_4e67e57c-d41c-4284-b991-dd9c2514421b.bai dst=null perm=null proto=rpc
17:37:54.917 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:37:54.918 WARN DFSUtil - Unexpected value for data transfer bytes=5949 duration=0
17:37:54.920 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:37:54.921 WARN DFSUtil - Unexpected value for data transfer bytes=32 duration=0
17:37:54.923 INFO Executor - Finished task 0.0 in stage 53.0 (TID 91). 989 bytes result sent to driver
17:37:54.923 INFO TaskSetManager - Finished task 0.0 in stage 53.0 (TID 91) in 123 ms on localhost (executor driver) (1/1)
17:37:54.924 INFO TaskSchedulerImpl - Removed TaskSet 53.0, whose tasks have all completed, from pool
17:37:54.924 INFO DAGScheduler - ResultStage 53 (count at ReadsSparkSinkUnitTest.java:185) finished in 0.146 s
17:37:54.924 INFO DAGScheduler - Job 39 is finished. Cancelling potential speculative or zombie tasks for this job
17:37:54.924 INFO TaskSchedulerImpl - Killing all running tasks in stage 53: Stage finished
17:37:54.924 INFO DAGScheduler - Job 39 finished: count at ReadsSparkSinkUnitTest.java:185, took 0.147194 s
17:37:54.925 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:37:54.929 INFO MemoryStore - Block broadcast_87 stored as values in memory (estimated size 298.0 KiB, free 1916.7 MiB)
17:37:54.935 INFO MemoryStore - Block broadcast_87_piece0 stored as bytes in memory (estimated size 50.3 KiB, free 1916.6 MiB)
17:37:54.935 INFO BlockManagerInfo - Added broadcast_87_piece0 in memory on localhost:37953 (size: 50.3 KiB, free: 1919.3 MiB)
17:37:54.936 INFO SparkContext - Created broadcast 87 from newAPIHadoopFile at PathSplitSource.java:96
17:37:54.956 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:37:54.965 INFO MemoryStore - Block broadcast_88 stored as values in memory (estimated size 298.0 KiB, free 1916.5 MiB)
17:37:54.966 INFO BlockManagerInfo - Removed broadcast_86_piece0 on localhost:37953 in memory (size: 153.6 KiB, free: 1919.5 MiB)
17:37:54.967 INFO BlockManagerInfo - Removed broadcast_85_piece0 on localhost:37953 in memory (size: 153.6 KiB, free: 1919.6 MiB)
17:37:54.968 INFO BlockManagerInfo - Removed broadcast_79_piece0 on localhost:37953 in memory (size: 9.6 KiB, free: 1919.6 MiB)
17:37:54.969 INFO BlockManagerInfo - Removed broadcast_84_piece0 on localhost:37953 in memory (size: 153.7 KiB, free: 1919.8 MiB)
17:37:54.970 INFO BlockManagerInfo - Removed broadcast_77_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.8 MiB)
17:37:54.971 INFO BlockManagerInfo - Removed broadcast_82_piece0 on localhost:37953 in memory (size: 67.1 KiB, free: 1919.9 MiB)
17:37:54.971 INFO MemoryStore - Block broadcast_88_piece0 stored as bytes in memory (estimated size 50.3 KiB, free 1918.8 MiB)
17:37:54.971 INFO BlockManagerInfo - Added broadcast_88_piece0 in memory on localhost:37953 (size: 50.3 KiB, free: 1919.8 MiB)
17:37:54.971 INFO SparkContext - Created broadcast 88 from newAPIHadoopFile at PathSplitSource.java:96
17:37:54.972 INFO BlockManagerInfo - Removed broadcast_83_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.9 MiB)
17:37:54.972 INFO BlockManagerInfo - Removed broadcast_80_piece0 on localhost:37953 in memory (size: 9.6 KiB, free: 1919.9 MiB)
17:37:54.991 INFO FileInputFormat - Total input files to process : 1
17:37:54.994 INFO MemoryStore - Block broadcast_89 stored as values in memory (estimated size 160.7 KiB, free 1919.2 MiB)
17:37:54.994 INFO MemoryStore - Block broadcast_89_piece0 stored as bytes in memory (estimated size 9.6 KiB, free 1919.2 MiB)
17:37:54.995 INFO BlockManagerInfo - Added broadcast_89_piece0 in memory on localhost:37953 (size: 9.6 KiB, free: 1919.9 MiB)
17:37:54.995 INFO SparkContext - Created broadcast 89 from broadcast at ReadsSparkSink.java:133
17:37:54.997 INFO MemoryStore - Block broadcast_90 stored as values in memory (estimated size 163.2 KiB, free 1919.0 MiB)
17:37:54.997 INFO MemoryStore - Block broadcast_90_piece0 stored as bytes in memory (estimated size 9.6 KiB, free 1919.0 MiB)
17:37:54.997 INFO BlockManagerInfo - Added broadcast_90_piece0 in memory on localhost:37953 (size: 9.6 KiB, free: 1919.9 MiB)
17:37:54.998 INFO SparkContext - Created broadcast 90 from broadcast at BamSink.java:76
17:37:55.000 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam.parts dst=null perm=null proto=rpc
17:37:55.001 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:37:55.001 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:37:55.001 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:37:55.002 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=mkdirs src=/user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam.parts/_temporary/0 dst=null perm=runner:supergroup:rwxr-xr-x proto=rpc
17:37:55.008 INFO SparkContext - Starting job: runJob at SparkHadoopWriter.scala:83
17:37:55.009 INFO DAGScheduler - Registering RDD 204 (mapToPair at SparkUtils.java:161) as input to shuffle 12
17:37:55.009 INFO DAGScheduler - Got job 40 (runJob at SparkHadoopWriter.scala:83) with 1 output partitions
17:37:55.009 INFO DAGScheduler - Final stage: ResultStage 55 (runJob at SparkHadoopWriter.scala:83)
17:37:55.009 INFO DAGScheduler - Parents of final stage: List(ShuffleMapStage 54)
17:37:55.009 INFO DAGScheduler - Missing parents: List(ShuffleMapStage 54)
17:37:55.009 INFO DAGScheduler - Submitting ShuffleMapStage 54 (MapPartitionsRDD[204] at mapToPair at SparkUtils.java:161), which has no missing parents
17:37:55.025 INFO MemoryStore - Block broadcast_91 stored as values in memory (estimated size 520.4 KiB, free 1918.5 MiB)
17:37:55.027 INFO MemoryStore - Block broadcast_91_piece0 stored as bytes in memory (estimated size 166.1 KiB, free 1918.3 MiB)
17:37:55.027 INFO BlockManagerInfo - Added broadcast_91_piece0 in memory on localhost:37953 (size: 166.1 KiB, free: 1919.7 MiB)
17:37:55.027 INFO SparkContext - Created broadcast 91 from broadcast at DAGScheduler.scala:1580
17:37:55.027 INFO DAGScheduler - Submitting 1 missing tasks from ShuffleMapStage 54 (MapPartitionsRDD[204] at mapToPair at SparkUtils.java:161) (first 15 tasks are for partitions Vector(0))
17:37:55.027 INFO TaskSchedulerImpl - Adding task set 54.0 with 1 tasks resource profile 0
17:37:55.028 INFO TaskSetManager - Starting task 0.0 in stage 54.0 (TID 92) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7901 bytes)
17:37:55.029 INFO Executor - Running task 0.0 in stage 54.0 (TID 92)
17:37:55.064 INFO NewHadoopRDD - Input split: file:/home/runner/work/gatk/gatk/src/test/resources/org/broadinstitute/hellbender/tools/BQSR/expected.HiSeq.1mb.1RG.2k_lines.alternate.recalibrated.DIQ.bam:0+216896
17:37:55.065 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:37:55.066 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:37:55.076 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:37:55.087 INFO Executor - Finished task 0.0 in stage 54.0 (TID 92). 1148 bytes result sent to driver
17:37:55.087 INFO TaskSetManager - Finished task 0.0 in stage 54.0 (TID 92) in 59 ms on localhost (executor driver) (1/1)
17:37:55.087 INFO TaskSchedulerImpl - Removed TaskSet 54.0, whose tasks have all completed, from pool
17:37:55.088 INFO DAGScheduler - ShuffleMapStage 54 (mapToPair at SparkUtils.java:161) finished in 0.078 s
17:37:55.088 INFO DAGScheduler - looking for newly runnable stages
17:37:55.088 INFO DAGScheduler - running: HashSet()
17:37:55.088 INFO DAGScheduler - waiting: HashSet(ResultStage 55)
17:37:55.088 INFO DAGScheduler - failed: HashSet()
17:37:55.088 INFO DAGScheduler - Submitting ResultStage 55 (MapPartitionsRDD[209] at mapToPair at BamSink.java:91), which has no missing parents
17:37:55.094 INFO MemoryStore - Block broadcast_92 stored as values in memory (estimated size 241.5 KiB, free 1918.1 MiB)
17:37:55.095 INFO MemoryStore - Block broadcast_92_piece0 stored as bytes in memory (estimated size 67.1 KiB, free 1918.0 MiB)
17:37:55.095 INFO BlockManagerInfo - Added broadcast_92_piece0 in memory on localhost:37953 (size: 67.1 KiB, free: 1919.7 MiB)
17:37:55.096 INFO SparkContext - Created broadcast 92 from broadcast at DAGScheduler.scala:1580
17:37:55.096 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 55 (MapPartitionsRDD[209] at mapToPair at BamSink.java:91) (first 15 tasks are for partitions Vector(0))
17:37:55.096 INFO TaskSchedulerImpl - Adding task set 55.0 with 1 tasks resource profile 0
17:37:55.097 INFO TaskSetManager - Starting task 0.0 in stage 55.0 (TID 93) (localhost, executor driver, partition 0, NODE_LOCAL, 7513 bytes)
17:37:55.097 INFO Executor - Running task 0.0 in stage 55.0 (TID 93)
17:37:55.103 INFO ShuffleBlockFetcherIterator - Getting 1 (343.8 KiB) non-empty blocks including 1 (343.8 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
17:37:55.103 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
17:37:55.118 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:37:55.118 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:37:55.118 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:37:55.118 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:37:55.118 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:37:55.118 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:37:55.119 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam.parts/_temporary/0/_temporary/attempt_202510271737556056307602572731118_0209_r_000000_0/part-r-00000 dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:37:55.120 WARN IntelDeflaterFactory - Intel Deflater not supported, using Java.util.zip.Deflater
17:37:55.121 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam.parts/_temporary/0/_temporary/attempt_202510271737556056307602572731118_0209_r_000000_0/.part-r-00000.sbi dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:37:55.122 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam.parts/_temporary/0/_temporary/attempt_202510271737556056307602572731118_0209_r_000000_0/.part-r-00000.bai dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:37:55.126 INFO StateChange - BLOCK* allocate blk_1073741852_1028, replicas=127.0.0.1:37927 for /user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam.parts/_temporary/0/_temporary/attempt_202510271737556056307602572731118_0209_r_000000_0/part-r-00000
17:37:55.128 INFO DataNode - Receiving BP-617366194-10.1.0.252-1761586664431:blk_1073741852_1028 src: /127.0.0.1:58026 dest: /127.0.0.1:37927
17:37:55.133 INFO clienttrace - src: /127.0.0.1:58026, dest: /127.0.0.1:37927, bytes: 230392, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1369393814_1, offset: 0, srvID: 379dd5e5-59c3-42cd-b9d3-67ca5e44560b, blockid: BP-617366194-10.1.0.252-1761586664431:blk_1073741852_1028, duration(ns): 4150086
17:37:55.133 INFO DataNode - PacketResponder: BP-617366194-10.1.0.252-1761586664431:blk_1073741852_1028, type=LAST_IN_PIPELINE terminating
17:37:55.133 INFO FSNamesystem - BLOCK* blk_1073741852_1028 is COMMITTED but not COMPLETE(numNodes= 0 < minimum = 1) in file /user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam.parts/_temporary/0/_temporary/attempt_202510271737556056307602572731118_0209_r_000000_0/part-r-00000
17:37:55.535 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam.parts/_temporary/0/_temporary/attempt_202510271737556056307602572731118_0209_r_000000_0/part-r-00000 is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:37:55.536 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam.parts/_temporary/0/_temporary/attempt_202510271737556056307602572731118_0209_r_000000_0/part-r-00000 dst=null perm=null proto=rpc
17:37:55.537 INFO StateChange - BLOCK* allocate blk_1073741853_1029, replicas=127.0.0.1:37927 for /user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam.parts/_temporary/0/_temporary/attempt_202510271737556056307602572731118_0209_r_000000_0/.part-r-00000.sbi
17:37:55.538 INFO DataNode - Receiving BP-617366194-10.1.0.252-1761586664431:blk_1073741853_1029 src: /127.0.0.1:58034 dest: /127.0.0.1:37927
17:37:55.539 INFO clienttrace - src: /127.0.0.1:58034, dest: /127.0.0.1:37927, bytes: 212, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1369393814_1, offset: 0, srvID: 379dd5e5-59c3-42cd-b9d3-67ca5e44560b, blockid: BP-617366194-10.1.0.252-1761586664431:blk_1073741853_1029, duration(ns): 536181
17:37:55.539 INFO DataNode - PacketResponder: BP-617366194-10.1.0.252-1761586664431:blk_1073741853_1029, type=LAST_IN_PIPELINE terminating
17:37:55.540 INFO FSNamesystem - BLOCK* blk_1073741853_1029 is COMMITTED but not COMPLETE(numNodes= 0 < minimum = 1) in file /user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam.parts/_temporary/0/_temporary/attempt_202510271737556056307602572731118_0209_r_000000_0/.part-r-00000.sbi
17:37:55.941 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam.parts/_temporary/0/_temporary/attempt_202510271737556056307602572731118_0209_r_000000_0/.part-r-00000.sbi is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:37:55.943 INFO StateChange - BLOCK* allocate blk_1073741854_1030, replicas=127.0.0.1:37927 for /user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam.parts/_temporary/0/_temporary/attempt_202510271737556056307602572731118_0209_r_000000_0/.part-r-00000.bai
17:37:55.944 INFO DataNode - Receiving BP-617366194-10.1.0.252-1761586664431:blk_1073741854_1030 src: /127.0.0.1:58048 dest: /127.0.0.1:37927
17:37:55.945 INFO clienttrace - src: /127.0.0.1:58048, dest: /127.0.0.1:37927, bytes: 5472, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1369393814_1, offset: 0, srvID: 379dd5e5-59c3-42cd-b9d3-67ca5e44560b, blockid: BP-617366194-10.1.0.252-1761586664431:blk_1073741854_1030, duration(ns): 552158
17:37:55.946 INFO DataNode - PacketResponder: BP-617366194-10.1.0.252-1761586664431:blk_1073741854_1030, type=LAST_IN_PIPELINE terminating
17:37:55.946 INFO FSNamesystem - BLOCK* blk_1073741854_1030 is COMMITTED but not COMPLETE(numNodes= 0 < minimum = 1) in file /user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam.parts/_temporary/0/_temporary/attempt_202510271737556056307602572731118_0209_r_000000_0/.part-r-00000.bai
17:37:56.348 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam.parts/_temporary/0/_temporary/attempt_202510271737556056307602572731118_0209_r_000000_0/.part-r-00000.bai is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:37:56.349 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam.parts/_temporary/0/_temporary/attempt_202510271737556056307602572731118_0209_r_000000_0 dst=null perm=null proto=rpc
17:37:56.350 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam.parts/_temporary/0/_temporary/attempt_202510271737556056307602572731118_0209_r_000000_0 dst=null perm=null proto=rpc
17:37:56.350 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam.parts/_temporary/0/task_202510271737556056307602572731118_0209_r_000000 dst=null perm=null proto=rpc
17:37:56.351 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=rename src=/user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam.parts/_temporary/0/_temporary/attempt_202510271737556056307602572731118_0209_r_000000_0 dst=/user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam.parts/_temporary/0/task_202510271737556056307602572731118_0209_r_000000 perm=runner:supergroup:rwxr-xr-x proto=rpc
17:37:56.351 INFO FileOutputCommitter - Saved output of task 'attempt_202510271737556056307602572731118_0209_r_000000_0' to hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam.parts/_temporary/0/task_202510271737556056307602572731118_0209_r_000000
17:37:56.351 INFO SparkHadoopMapRedUtil - attempt_202510271737556056307602572731118_0209_r_000000_0: Committed. Elapsed time: 2 ms.
17:37:56.352 INFO Executor - Finished task 0.0 in stage 55.0 (TID 93). 1858 bytes result sent to driver
17:37:56.353 INFO TaskSetManager - Finished task 0.0 in stage 55.0 (TID 93) in 1256 ms on localhost (executor driver) (1/1)
17:37:56.353 INFO TaskSchedulerImpl - Removed TaskSet 55.0, whose tasks have all completed, from pool
17:37:56.353 INFO DAGScheduler - ResultStage 55 (runJob at SparkHadoopWriter.scala:83) finished in 1.265 s
17:37:56.353 INFO DAGScheduler - Job 40 is finished. Cancelling potential speculative or zombie tasks for this job
17:37:56.353 INFO TaskSchedulerImpl - Killing all running tasks in stage 55: Stage finished
17:37:56.353 INFO DAGScheduler - Job 40 finished: runJob at SparkHadoopWriter.scala:83, took 1.345105 s
17:37:56.354 INFO SparkHadoopWriter - Start to commit write Job job_202510271737556056307602572731118_0209.
17:37:56.354 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=listStatus src=/user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam.parts/_temporary/0 dst=null perm=null proto=rpc
17:37:56.355 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam.parts dst=null perm=null proto=rpc
17:37:56.355 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=listStatus src=/user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam.parts/_temporary/0/task_202510271737556056307602572731118_0209_r_000000 dst=null perm=null proto=rpc
17:37:56.356 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam.parts/.part-r-00000.bai dst=null perm=null proto=rpc
17:37:56.357 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=rename src=/user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam.parts/_temporary/0/task_202510271737556056307602572731118_0209_r_000000/.part-r-00000.bai dst=/user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam.parts/.part-r-00000.bai perm=runner:supergroup:rw-r--r-- proto=rpc
17:37:56.357 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam.parts/.part-r-00000.sbi dst=null perm=null proto=rpc
17:37:56.358 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=rename src=/user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam.parts/_temporary/0/task_202510271737556056307602572731118_0209_r_000000/.part-r-00000.sbi dst=/user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam.parts/.part-r-00000.sbi perm=runner:supergroup:rw-r--r-- proto=rpc
17:37:56.359 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam.parts/part-r-00000 dst=null perm=null proto=rpc
17:37:56.359 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=rename src=/user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam.parts/_temporary/0/task_202510271737556056307602572731118_0209_r_000000/part-r-00000 dst=/user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam.parts/part-r-00000 perm=runner:supergroup:rw-r--r-- proto=rpc
17:37:56.360 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=delete src=/user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam.parts/_temporary dst=null perm=null proto=rpc
17:37:56.361 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam.parts/_SUCCESS dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:37:56.362 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam.parts/_SUCCESS is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:37:56.363 INFO audit - allowed=false ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=delete src=/user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam.parts/.spark-staging-209 dst=null perm=null proto=rpc
17:37:56.363 INFO SparkHadoopWriter - Write Job job_202510271737556056307602572731118_0209 committed. Elapsed time: 9 ms.
17:37:56.364 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam.parts/header dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:37:56.364 WARN IntelDeflaterFactory - Intel Deflater not supported, using Java.util.zip.Deflater
17:37:56.365 INFO StateChange - BLOCK* allocate blk_1073741855_1031, replicas=127.0.0.1:37927 for /user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam.parts/header
17:37:56.367 INFO DataNode - Receiving BP-617366194-10.1.0.252-1761586664431:blk_1073741855_1031 src: /127.0.0.1:58052 dest: /127.0.0.1:37927
17:37:56.368 INFO clienttrace - src: /127.0.0.1:58052, dest: /127.0.0.1:37927, bytes: 5901, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1369393814_1, offset: 0, srvID: 379dd5e5-59c3-42cd-b9d3-67ca5e44560b, blockid: BP-617366194-10.1.0.252-1761586664431:blk_1073741855_1031, duration(ns): 616223
17:37:56.368 INFO DataNode - PacketResponder: BP-617366194-10.1.0.252-1761586664431:blk_1073741855_1031, type=LAST_IN_PIPELINE terminating
17:37:56.369 INFO FSNamesystem - BLOCK* blk_1073741855_1031 is COMMITTED but not COMPLETE(numNodes= 0 < minimum = 1) in file /user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam.parts/header
17:37:56.770 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam.parts/header is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:37:56.771 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam.parts/terminator dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:37:56.772 INFO StateChange - BLOCK* allocate blk_1073741856_1032, replicas=127.0.0.1:37927 for /user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam.parts/terminator
17:37:56.774 INFO DataNode - Receiving BP-617366194-10.1.0.252-1761586664431:blk_1073741856_1032 src: /127.0.0.1:58058 dest: /127.0.0.1:37927
17:37:56.775 INFO clienttrace - src: /127.0.0.1:58058, dest: /127.0.0.1:37927, bytes: 28, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1369393814_1, offset: 0, srvID: 379dd5e5-59c3-42cd-b9d3-67ca5e44560b, blockid: BP-617366194-10.1.0.252-1761586664431:blk_1073741856_1032, duration(ns): 533973
17:37:56.775 INFO DataNode - PacketResponder: BP-617366194-10.1.0.252-1761586664431:blk_1073741856_1032, type=LAST_IN_PIPELINE terminating
17:37:56.776 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam.parts/terminator is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:37:56.777 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=listStatus src=/user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam.parts dst=null perm=null proto=rpc
17:37:56.778 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam.parts/output dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:37:56.778 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam.parts/output is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:37:56.779 INFO HadoopFileSystemWrapper - Concatenating 3 parts to hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam
17:37:56.779 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=concat src=[/user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam.parts/header, /user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam.parts/part-r-00000, /user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam.parts/terminator] dst=/user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam.parts/output perm=runner:supergroup:rw-r--r-- proto=rpc
17:37:56.780 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam dst=null perm=null proto=rpc
17:37:56.781 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=rename src=/user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam.parts/output dst=/user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam perm=runner:supergroup:rw-r--r-- proto=rpc
17:37:56.781 INFO HadoopFileSystemWrapper - Concatenating to hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam done
17:37:56.781 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam dst=null perm=null proto=rpc
17:37:56.782 INFO IndexFileMerger - Merging .sbi files in temp directory hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam.parts/ to hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam.sbi
17:37:56.782 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=listStatus src=/user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam.parts dst=null perm=null proto=rpc
17:37:56.783 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam.sbi dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:37:56.784 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam.parts/.part-r-00000.sbi dst=null perm=null proto=rpc
17:37:56.785 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam.parts/.part-r-00000.sbi dst=null perm=null proto=rpc
17:37:56.786 WARN DFSUtil - Unexpected value for data transfer bytes=216 duration=0
17:37:56.786 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=delete src=/user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam.parts/.part-r-00000.sbi dst=null perm=null proto=rpc
17:37:56.787 INFO StateChange - BLOCK* allocate blk_1073741857_1033, replicas=127.0.0.1:37927 for /user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam.sbi
17:37:56.788 INFO DataNode - Receiving BP-617366194-10.1.0.252-1761586664431:blk_1073741857_1033 src: /127.0.0.1:58072 dest: /127.0.0.1:37927
17:37:56.789 INFO clienttrace - src: /127.0.0.1:58072, dest: /127.0.0.1:37927, bytes: 212, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1369393814_1, offset: 0, srvID: 379dd5e5-59c3-42cd-b9d3-67ca5e44560b, blockid: BP-617366194-10.1.0.252-1761586664431:blk_1073741857_1033, duration(ns): 480043
17:37:56.790 INFO DataNode - PacketResponder: BP-617366194-10.1.0.252-1761586664431:blk_1073741857_1033, type=LAST_IN_PIPELINE terminating
17:37:56.790 INFO FSNamesystem - BLOCK* blk_1073741857_1033 is COMMITTED but not COMPLETE(numNodes= 0 < minimum = 1) in file /user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam.sbi
17:37:57.191 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam.sbi is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:37:57.192 INFO IndexFileMerger - Done merging .sbi files
17:37:57.192 INFO IndexFileMerger - Merging .bai files in temp directory hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam.parts/ to hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam.bai
17:37:57.192 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=listStatus src=/user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam.parts dst=null perm=null proto=rpc
17:37:57.194 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam.bai dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:37:57.195 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam.parts/.part-r-00000.bai dst=null perm=null proto=rpc
17:37:57.195 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam.parts/.part-r-00000.bai dst=null perm=null proto=rpc
17:37:57.197 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=delete src=/user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam.parts/.part-r-00000.bai dst=null perm=null proto=rpc
17:37:57.199 INFO StateChange - BLOCK* allocate blk_1073741858_1034, replicas=127.0.0.1:37927 for /user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam.bai
17:37:57.200 INFO DataNode - Receiving BP-617366194-10.1.0.252-1761586664431:blk_1073741858_1034 src: /127.0.0.1:58082 dest: /127.0.0.1:37927
17:37:57.201 INFO clienttrace - src: /127.0.0.1:58082, dest: /127.0.0.1:37927, bytes: 5472, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1369393814_1, offset: 0, srvID: 379dd5e5-59c3-42cd-b9d3-67ca5e44560b, blockid: BP-617366194-10.1.0.252-1761586664431:blk_1073741858_1034, duration(ns): 560321
17:37:57.201 INFO DataNode - PacketResponder: BP-617366194-10.1.0.252-1761586664431:blk_1073741858_1034, type=LAST_IN_PIPELINE terminating
17:37:57.202 INFO FSNamesystem - BLOCK* blk_1073741858_1034 is COMMITTED but not COMPLETE(numNodes= 0 < minimum = 1) in file /user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam.bai
17:37:57.599 INFO FsDatasetAsyncDiskService - Scheduling blk_1073741853_1029 replica FinalizedReplica, blk_1073741853_1029, FINALIZED
getNumBytes() = 212
getBytesOnDisk() = 212
getVisibleLength()= 212
getVolume() = /tmp/minicluster_storage14473397491642437130/data/data1
getBlockURI() = file:/tmp/minicluster_storage14473397491642437130/data/data1/current/BP-617366194-10.1.0.252-1761586664431/current/finalized/subdir0/subdir0/blk_1073741853 for deletion
17:37:57.599 INFO FsDatasetAsyncDiskService - Deleted BP-617366194-10.1.0.252-1761586664431 blk_1073741853_1029 URI file:/tmp/minicluster_storage14473397491642437130/data/data1/current/BP-617366194-10.1.0.252-1761586664431/current/finalized/subdir0/subdir0/blk_1073741853
17:37:57.603 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam.bai is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:37:57.603 INFO IndexFileMerger - Done merging .bai files
17:37:57.604 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=delete src=/user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam.parts dst=null perm=null proto=rpc
17:37:57.613 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam.bai dst=null perm=null proto=rpc
17:37:57.620 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam.sbi dst=null perm=null proto=rpc
17:37:57.621 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam.sbi dst=null perm=null proto=rpc
17:37:57.622 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam.sbi dst=null perm=null proto=rpc
17:37:57.623 WARN DFSUtil - Unexpected value for data transfer bytes=216 duration=0
17:37:57.623 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam dst=null perm=null proto=rpc
17:37:57.625 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam dst=null perm=null proto=rpc
17:37:57.626 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam dst=null perm=null proto=rpc
17:37:57.626 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam dst=null perm=null proto=rpc
17:37:57.627 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam.bai dst=null perm=null proto=rpc
17:37:57.628 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam.bai dst=null perm=null proto=rpc
17:37:57.628 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam.bai dst=null perm=null proto=rpc
17:37:57.629 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:37:57.630 WARN DFSUtil - Unexpected value for data transfer bytes=5949 duration=0
17:37:57.632 WARN DFSUtil - Unexpected value for data transfer bytes=5516 duration=0
17:37:57.633 WARN DFSUtil - Unexpected value for data transfer bytes=5516 duration=0
17:37:57.633 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam.sbi dst=null perm=null proto=rpc
17:37:57.634 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam.sbi dst=null perm=null proto=rpc
17:37:57.634 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam.sbi dst=null perm=null proto=rpc
17:37:57.635 WARN DFSUtil - Unexpected value for data transfer bytes=216 duration=0
17:37:57.636 INFO MemoryStore - Block broadcast_93 stored as values in memory (estimated size 320.0 B, free 1918.0 MiB)
17:37:57.636 INFO MemoryStore - Block broadcast_93_piece0 stored as bytes in memory (estimated size 233.0 B, free 1918.0 MiB)
17:37:57.636 INFO BlockManagerInfo - Added broadcast_93_piece0 in memory on localhost:37953 (size: 233.0 B, free: 1919.7 MiB)
17:37:57.637 INFO SparkContext - Created broadcast 93 from broadcast at BamSource.java:104
17:37:57.638 INFO MemoryStore - Block broadcast_94 stored as values in memory (estimated size 297.9 KiB, free 1917.7 MiB)
17:37:57.643 INFO MemoryStore - Block broadcast_94_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1917.7 MiB)
17:37:57.643 INFO BlockManagerInfo - Added broadcast_94_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.6 MiB)
17:37:57.644 INFO SparkContext - Created broadcast 94 from newAPIHadoopFile at PathSplitSource.java:96
17:37:57.653 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam dst=null perm=null proto=rpc
17:37:57.654 INFO FileInputFormat - Total input files to process : 1
17:37:57.654 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam dst=null perm=null proto=rpc
17:37:57.668 INFO SparkContext - Starting job: collect at ReadsSparkSinkUnitTest.java:182
17:37:57.668 INFO DAGScheduler - Got job 41 (collect at ReadsSparkSinkUnitTest.java:182) with 1 output partitions
17:37:57.668 INFO DAGScheduler - Final stage: ResultStage 56 (collect at ReadsSparkSinkUnitTest.java:182)
17:37:57.668 INFO DAGScheduler - Parents of final stage: List()
17:37:57.668 INFO DAGScheduler - Missing parents: List()
17:37:57.669 INFO DAGScheduler - Submitting ResultStage 56 (MapPartitionsRDD[215] at filter at ReadsSparkSource.java:96), which has no missing parents
17:37:57.675 INFO MemoryStore - Block broadcast_95 stored as values in memory (estimated size 148.2 KiB, free 1917.5 MiB)
17:37:57.676 INFO MemoryStore - Block broadcast_95_piece0 stored as bytes in memory (estimated size 54.6 KiB, free 1917.5 MiB)
17:37:57.676 INFO BlockManagerInfo - Added broadcast_95_piece0 in memory on localhost:37953 (size: 54.6 KiB, free: 1919.6 MiB)
17:37:57.676 INFO SparkContext - Created broadcast 95 from broadcast at DAGScheduler.scala:1580
17:37:57.676 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 56 (MapPartitionsRDD[215] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:37:57.676 INFO TaskSchedulerImpl - Adding task set 56.0 with 1 tasks resource profile 0
17:37:57.677 INFO TaskSetManager - Starting task 0.0 in stage 56.0 (TID 94) (localhost, executor driver, partition 0, ANY, 7852 bytes)
17:37:57.677 INFO Executor - Running task 0.0 in stage 56.0 (TID 94)
17:37:57.690 INFO NewHadoopRDD - Input split: hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam:0+236321
17:37:57.691 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam dst=null perm=null proto=rpc
17:37:57.691 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam dst=null perm=null proto=rpc
17:37:57.692 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam.bai dst=null perm=null proto=rpc
17:37:57.693 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam.bai dst=null perm=null proto=rpc
17:37:57.694 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam.bai dst=null perm=null proto=rpc
17:37:57.695 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:37:57.695 WARN DFSUtil - Unexpected value for data transfer bytes=5949 duration=0
17:37:57.697 WARN DFSUtil - Unexpected value for data transfer bytes=5516 duration=0
17:37:57.698 WARN DFSUtil - Unexpected value for data transfer bytes=5516 duration=0
17:37:57.699 WARN DFSUtil - Unexpected value for data transfer bytes=232192 duration=0
17:37:57.700 WARN DFSUtil - Unexpected value for data transfer bytes=32 duration=0
17:37:57.702 INFO Executor - Finished task 0.0 in stage 56.0 (TID 94). 650141 bytes result sent to driver
17:37:57.704 INFO TaskSetManager - Finished task 0.0 in stage 56.0 (TID 94) in 27 ms on localhost (executor driver) (1/1)
17:37:57.704 INFO TaskSchedulerImpl - Removed TaskSet 56.0, whose tasks have all completed, from pool
17:37:57.704 INFO DAGScheduler - ResultStage 56 (collect at ReadsSparkSinkUnitTest.java:182) finished in 0.034 s
17:37:57.704 INFO DAGScheduler - Job 41 is finished. Cancelling potential speculative or zombie tasks for this job
17:37:57.704 INFO TaskSchedulerImpl - Killing all running tasks in stage 56: Stage finished
17:37:57.704 INFO DAGScheduler - Job 41 finished: collect at ReadsSparkSinkUnitTest.java:182, took 0.036473 s
17:37:57.714 INFO SparkContext - Starting job: count at ReadsSparkSinkUnitTest.java:185
17:37:57.715 INFO DAGScheduler - Got job 42 (count at ReadsSparkSinkUnitTest.java:185) with 1 output partitions
17:37:57.715 INFO DAGScheduler - Final stage: ResultStage 57 (count at ReadsSparkSinkUnitTest.java:185)
17:37:57.715 INFO DAGScheduler - Parents of final stage: List()
17:37:57.715 INFO DAGScheduler - Missing parents: List()
17:37:57.715 INFO DAGScheduler - Submitting ResultStage 57 (MapPartitionsRDD[197] at filter at ReadsSparkSource.java:96), which has no missing parents
17:37:57.731 INFO MemoryStore - Block broadcast_96 stored as values in memory (estimated size 426.1 KiB, free 1917.1 MiB)
17:37:57.737 INFO BlockManagerInfo - Removed broadcast_95_piece0 on localhost:37953 in memory (size: 54.6 KiB, free: 1919.6 MiB)
17:37:57.738 INFO MemoryStore - Block broadcast_96_piece0 stored as bytes in memory (estimated size 153.6 KiB, free 1917.1 MiB)
17:37:57.738 INFO BlockManagerInfo - Added broadcast_96_piece0 in memory on localhost:37953 (size: 153.6 KiB, free: 1919.5 MiB)
17:37:57.738 INFO BlockManagerInfo - Removed broadcast_92_piece0 on localhost:37953 in memory (size: 67.1 KiB, free: 1919.5 MiB)
17:37:57.738 INFO SparkContext - Created broadcast 96 from broadcast at DAGScheduler.scala:1580
17:37:57.738 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 57 (MapPartitionsRDD[197] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:37:57.738 INFO TaskSchedulerImpl - Adding task set 57.0 with 1 tasks resource profile 0
17:37:57.739 INFO BlockManagerInfo - Removed broadcast_88_piece0 on localhost:37953 in memory (size: 50.3 KiB, free: 1919.6 MiB)
17:37:57.739 INFO TaskSetManager - Starting task 0.0 in stage 57.0 (TID 95) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7912 bytes)
17:37:57.740 INFO Executor - Running task 0.0 in stage 57.0 (TID 95)
17:37:57.740 INFO BlockManagerInfo - Removed broadcast_89_piece0 on localhost:37953 in memory (size: 9.6 KiB, free: 1919.6 MiB)
17:37:57.741 INFO BlockManagerInfo - Removed broadcast_90_piece0 on localhost:37953 in memory (size: 9.6 KiB, free: 1919.6 MiB)
17:37:57.742 INFO BlockManagerInfo - Removed broadcast_91_piece0 on localhost:37953 in memory (size: 166.1 KiB, free: 1919.8 MiB)
17:37:57.773 INFO NewHadoopRDD - Input split: file:/home/runner/work/gatk/gatk/src/test/resources/org/broadinstitute/hellbender/tools/BQSR/expected.HiSeq.1mb.1RG.2k_lines.alternate.recalibrated.DIQ.bam:0+216896
17:37:57.774 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:37:57.775 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:37:57.785 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:37:57.789 INFO Executor - Finished task 0.0 in stage 57.0 (TID 95). 989 bytes result sent to driver
17:37:57.789 INFO TaskSetManager - Finished task 0.0 in stage 57.0 (TID 95) in 50 ms on localhost (executor driver) (1/1)
17:37:57.789 INFO TaskSchedulerImpl - Removed TaskSet 57.0, whose tasks have all completed, from pool
17:37:57.790 INFO DAGScheduler - ResultStage 57 (count at ReadsSparkSinkUnitTest.java:185) finished in 0.075 s
17:37:57.790 INFO DAGScheduler - Job 42 is finished. Cancelling potential speculative or zombie tasks for this job
17:37:57.790 INFO TaskSchedulerImpl - Killing all running tasks in stage 57: Stage finished
17:37:57.790 INFO DAGScheduler - Job 42 finished: count at ReadsSparkSinkUnitTest.java:185, took 0.075414 s
17:37:57.793 INFO SparkContext - Starting job: count at ReadsSparkSinkUnitTest.java:185
17:37:57.794 INFO DAGScheduler - Got job 43 (count at ReadsSparkSinkUnitTest.java:185) with 1 output partitions
17:37:57.794 INFO DAGScheduler - Final stage: ResultStage 58 (count at ReadsSparkSinkUnitTest.java:185)
17:37:57.794 INFO DAGScheduler - Parents of final stage: List()
17:37:57.794 INFO DAGScheduler - Missing parents: List()
17:37:57.794 INFO DAGScheduler - Submitting ResultStage 58 (MapPartitionsRDD[215] at filter at ReadsSparkSource.java:96), which has no missing parents
17:37:57.800 INFO MemoryStore - Block broadcast_97 stored as values in memory (estimated size 148.1 KiB, free 1918.6 MiB)
17:37:57.800 INFO MemoryStore - Block broadcast_97_piece0 stored as bytes in memory (estimated size 54.6 KiB, free 1918.6 MiB)
17:37:57.801 INFO BlockManagerInfo - Added broadcast_97_piece0 in memory on localhost:37953 (size: 54.6 KiB, free: 1919.7 MiB)
17:37:57.801 INFO SparkContext - Created broadcast 97 from broadcast at DAGScheduler.scala:1580
17:37:57.801 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 58 (MapPartitionsRDD[215] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:37:57.801 INFO TaskSchedulerImpl - Adding task set 58.0 with 1 tasks resource profile 0
17:37:57.802 INFO TaskSetManager - Starting task 0.0 in stage 58.0 (TID 96) (localhost, executor driver, partition 0, ANY, 7852 bytes)
17:37:57.802 INFO Executor - Running task 0.0 in stage 58.0 (TID 96)
17:37:57.814 INFO NewHadoopRDD - Input split: hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam:0+236321
17:37:57.815 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam dst=null perm=null proto=rpc
17:37:57.816 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam dst=null perm=null proto=rpc
17:37:57.817 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam.bai dst=null perm=null proto=rpc
17:37:57.818 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam.bai dst=null perm=null proto=rpc
17:37:57.819 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest2_03021249-7d82-457e-8872-7c801957601f.bam.bai dst=null perm=null proto=rpc
17:37:57.820 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:37:57.822 WARN DFSUtil - Unexpected value for data transfer bytes=5516 duration=0
17:37:57.823 WARN DFSUtil - Unexpected value for data transfer bytes=5516 duration=0
17:37:57.824 WARN DFSUtil - Unexpected value for data transfer bytes=232192 duration=0
17:37:57.824 WARN DFSUtil - Unexpected value for data transfer bytes=32 duration=0
17:37:57.826 INFO Executor - Finished task 0.0 in stage 58.0 (TID 96). 989 bytes result sent to driver
17:37:57.827 INFO TaskSetManager - Finished task 0.0 in stage 58.0 (TID 96) in 25 ms on localhost (executor driver) (1/1)
17:37:57.827 INFO TaskSchedulerImpl - Removed TaskSet 58.0, whose tasks have all completed, from pool
17:37:57.827 INFO DAGScheduler - ResultStage 58 (count at ReadsSparkSinkUnitTest.java:185) finished in 0.033 s
17:37:57.827 INFO DAGScheduler - Job 43 is finished. Cancelling potential speculative or zombie tasks for this job
17:37:57.827 INFO TaskSchedulerImpl - Killing all running tasks in stage 58: Stage finished
17:37:57.827 INFO DAGScheduler - Job 43 finished: count at ReadsSparkSinkUnitTest.java:185, took 0.033983 s
17:37:57.829 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:37:57.832 INFO MemoryStore - Block broadcast_98 stored as values in memory (estimated size 298.0 KiB, free 1918.3 MiB)
17:37:57.837 INFO MemoryStore - Block broadcast_98_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1918.2 MiB)
17:37:57.837 INFO BlockManagerInfo - Added broadcast_98_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.6 MiB)
17:37:57.838 INFO SparkContext - Created broadcast 98 from newAPIHadoopFile at PathSplitSource.java:96
17:37:57.858 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:37:57.860 INFO MemoryStore - Block broadcast_99 stored as values in memory (estimated size 298.0 KiB, free 1917.9 MiB)
17:37:57.865 INFO MemoryStore - Block broadcast_99_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1917.9 MiB)
17:37:57.865 INFO BlockManagerInfo - Added broadcast_99_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.6 MiB)
17:37:57.866 INFO SparkContext - Created broadcast 99 from newAPIHadoopFile at PathSplitSource.java:96
17:37:57.885 INFO FileInputFormat - Total input files to process : 1
17:37:57.887 INFO MemoryStore - Block broadcast_100 stored as values in memory (estimated size 19.6 KiB, free 1917.9 MiB)
17:37:57.888 INFO MemoryStore - Block broadcast_100_piece0 stored as bytes in memory (estimated size 1890.0 B, free 1917.9 MiB)
17:37:57.888 INFO BlockManagerInfo - Added broadcast_100_piece0 in memory on localhost:37953 (size: 1890.0 B, free: 1919.6 MiB)
17:37:57.888 INFO SparkContext - Created broadcast 100 from broadcast at ReadsSparkSink.java:133
17:37:57.889 INFO MemoryStore - Block broadcast_101 stored as values in memory (estimated size 20.0 KiB, free 1917.8 MiB)
17:37:57.889 INFO MemoryStore - Block broadcast_101_piece0 stored as bytes in memory (estimated size 1890.0 B, free 1917.8 MiB)
17:37:57.890 INFO BlockManagerInfo - Added broadcast_101_piece0 in memory on localhost:37953 (size: 1890.0 B, free: 1919.6 MiB)
17:37:57.890 INFO SparkContext - Created broadcast 101 from broadcast at BamSink.java:76
17:37:57.893 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam.parts dst=null perm=null proto=rpc
17:37:57.893 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:37:57.893 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:37:57.893 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:37:57.894 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=mkdirs src=/user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam.parts/_temporary/0 dst=null perm=runner:supergroup:rwxr-xr-x proto=rpc
17:37:57.900 INFO SparkContext - Starting job: runJob at SparkHadoopWriter.scala:83
17:37:57.900 INFO DAGScheduler - Registering RDD 229 (mapToPair at SparkUtils.java:161) as input to shuffle 13
17:37:57.901 INFO DAGScheduler - Got job 44 (runJob at SparkHadoopWriter.scala:83) with 1 output partitions
17:37:57.901 INFO DAGScheduler - Final stage: ResultStage 60 (runJob at SparkHadoopWriter.scala:83)
17:37:57.901 INFO DAGScheduler - Parents of final stage: List(ShuffleMapStage 59)
17:37:57.901 INFO DAGScheduler - Missing parents: List(ShuffleMapStage 59)
17:37:57.901 INFO DAGScheduler - Submitting ShuffleMapStage 59 (MapPartitionsRDD[229] at mapToPair at SparkUtils.java:161), which has no missing parents
17:37:57.916 INFO MemoryStore - Block broadcast_102 stored as values in memory (estimated size 434.3 KiB, free 1917.4 MiB)
17:37:57.918 INFO MemoryStore - Block broadcast_102_piece0 stored as bytes in memory (estimated size 157.6 KiB, free 1917.3 MiB)
17:37:57.918 INFO BlockManagerInfo - Added broadcast_102_piece0 in memory on localhost:37953 (size: 157.6 KiB, free: 1919.4 MiB)
17:37:57.918 INFO SparkContext - Created broadcast 102 from broadcast at DAGScheduler.scala:1580
17:37:57.918 INFO DAGScheduler - Submitting 1 missing tasks from ShuffleMapStage 59 (MapPartitionsRDD[229] at mapToPair at SparkUtils.java:161) (first 15 tasks are for partitions Vector(0))
17:37:57.918 INFO TaskSchedulerImpl - Adding task set 59.0 with 1 tasks resource profile 0
17:37:57.919 INFO TaskSetManager - Starting task 0.0 in stage 59.0 (TID 97) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7882 bytes)
17:37:57.920 INFO Executor - Running task 0.0 in stage 59.0 (TID 97)
17:37:57.954 INFO NewHadoopRDD - Input split: file:/home/runner/work/gatk/gatk/src/test/resources/org/broadinstitute/hellbender/tools/BQSR/CEUTrio.HiSeq.WGS.b37.ch20.1m-1m1k.NA12878.bam:0+211123
17:37:57.955 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:37:57.956 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:37:57.962 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:37:57.972 INFO Executor - Finished task 0.0 in stage 59.0 (TID 97). 1148 bytes result sent to driver
17:37:57.972 INFO TaskSetManager - Finished task 0.0 in stage 59.0 (TID 97) in 53 ms on localhost (executor driver) (1/1)
17:37:57.972 INFO TaskSchedulerImpl - Removed TaskSet 59.0, whose tasks have all completed, from pool
17:37:57.973 INFO DAGScheduler - ShuffleMapStage 59 (mapToPair at SparkUtils.java:161) finished in 0.072 s
17:37:57.973 INFO DAGScheduler - looking for newly runnable stages
17:37:57.973 INFO DAGScheduler - running: HashSet()
17:37:57.973 INFO DAGScheduler - waiting: HashSet(ResultStage 60)
17:37:57.973 INFO DAGScheduler - failed: HashSet()
17:37:57.973 INFO DAGScheduler - Submitting ResultStage 60 (MapPartitionsRDD[234] at mapToPair at BamSink.java:91), which has no missing parents
17:37:57.979 INFO MemoryStore - Block broadcast_103 stored as values in memory (estimated size 155.4 KiB, free 1917.1 MiB)
17:37:57.980 INFO MemoryStore - Block broadcast_103_piece0 stored as bytes in memory (estimated size 58.5 KiB, free 1917.0 MiB)
17:37:57.980 INFO BlockManagerInfo - Added broadcast_103_piece0 in memory on localhost:37953 (size: 58.5 KiB, free: 1919.4 MiB)
17:37:57.980 INFO SparkContext - Created broadcast 103 from broadcast at DAGScheduler.scala:1580
17:37:57.981 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 60 (MapPartitionsRDD[234] at mapToPair at BamSink.java:91) (first 15 tasks are for partitions Vector(0))
17:37:57.981 INFO TaskSchedulerImpl - Adding task set 60.0 with 1 tasks resource profile 0
17:37:57.981 INFO TaskSetManager - Starting task 0.0 in stage 60.0 (TID 98) (localhost, executor driver, partition 0, NODE_LOCAL, 7513 bytes)
17:37:57.982 INFO Executor - Running task 0.0 in stage 60.0 (TID 98)
17:37:57.987 INFO ShuffleBlockFetcherIterator - Getting 1 (312.6 KiB) non-empty blocks including 1 (312.6 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
17:37:57.987 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
17:37:58.002 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:37:58.002 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:37:58.002 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:37:58.002 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:37:58.002 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:37:58.002 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:37:58.004 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam.parts/_temporary/0/_temporary/attempt_202510271737578168487553981124570_0234_r_000000_0/part-r-00000 dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:37:58.004 WARN IntelDeflaterFactory - Intel Deflater not supported, using Java.util.zip.Deflater
17:37:58.005 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam.parts/_temporary/0/_temporary/attempt_202510271737578168487553981124570_0234_r_000000_0/.part-r-00000.sbi dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:37:58.006 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam.parts/_temporary/0/_temporary/attempt_202510271737578168487553981124570_0234_r_000000_0/.part-r-00000.bai dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:37:58.010 INFO StateChange - BLOCK* allocate blk_1073741859_1035, replicas=127.0.0.1:37927 for /user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam.parts/_temporary/0/_temporary/attempt_202510271737578168487553981124570_0234_r_000000_0/part-r-00000
17:37:58.012 INFO DataNode - Receiving BP-617366194-10.1.0.252-1761586664431:blk_1073741859_1035 src: /127.0.0.1:54542 dest: /127.0.0.1:37927
17:37:58.016 INFO clienttrace - src: /127.0.0.1:54542, dest: /127.0.0.1:37927, bytes: 235870, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1369393814_1, offset: 0, srvID: 379dd5e5-59c3-42cd-b9d3-67ca5e44560b, blockid: BP-617366194-10.1.0.252-1761586664431:blk_1073741859_1035, duration(ns): 3484601
17:37:58.016 INFO DataNode - PacketResponder: BP-617366194-10.1.0.252-1761586664431:blk_1073741859_1035, type=LAST_IN_PIPELINE terminating
17:37:58.017 INFO FSNamesystem - BLOCK* blk_1073741859_1035 is COMMITTED but not COMPLETE(numNodes= 0 < minimum = 1) in file /user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam.parts/_temporary/0/_temporary/attempt_202510271737578168487553981124570_0234_r_000000_0/part-r-00000
17:37:58.418 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam.parts/_temporary/0/_temporary/attempt_202510271737578168487553981124570_0234_r_000000_0/part-r-00000 is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:37:58.419 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam.parts/_temporary/0/_temporary/attempt_202510271737578168487553981124570_0234_r_000000_0/part-r-00000 dst=null perm=null proto=rpc
17:37:58.420 INFO StateChange - BLOCK* allocate blk_1073741860_1036, replicas=127.0.0.1:37927 for /user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam.parts/_temporary/0/_temporary/attempt_202510271737578168487553981124570_0234_r_000000_0/.part-r-00000.sbi
17:37:58.421 INFO DataNode - Receiving BP-617366194-10.1.0.252-1761586664431:blk_1073741860_1036 src: /127.0.0.1:54544 dest: /127.0.0.1:37927
17:37:58.423 INFO clienttrace - src: /127.0.0.1:54544, dest: /127.0.0.1:37927, bytes: 204, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1369393814_1, offset: 0, srvID: 379dd5e5-59c3-42cd-b9d3-67ca5e44560b, blockid: BP-617366194-10.1.0.252-1761586664431:blk_1073741860_1036, duration(ns): 588178
17:37:58.423 INFO DataNode - PacketResponder: BP-617366194-10.1.0.252-1761586664431:blk_1073741860_1036, type=LAST_IN_PIPELINE terminating
17:37:58.423 INFO FSNamesystem - BLOCK* blk_1073741860_1036 is COMMITTED but not COMPLETE(numNodes= 0 < minimum = 1) in file /user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam.parts/_temporary/0/_temporary/attempt_202510271737578168487553981124570_0234_r_000000_0/.part-r-00000.sbi
17:37:58.825 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam.parts/_temporary/0/_temporary/attempt_202510271737578168487553981124570_0234_r_000000_0/.part-r-00000.sbi is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:37:58.826 INFO StateChange - BLOCK* allocate blk_1073741861_1037, replicas=127.0.0.1:37927 for /user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam.parts/_temporary/0/_temporary/attempt_202510271737578168487553981124570_0234_r_000000_0/.part-r-00000.bai
17:37:58.827 INFO DataNode - Receiving BP-617366194-10.1.0.252-1761586664431:blk_1073741861_1037 src: /127.0.0.1:54546 dest: /127.0.0.1:37927
17:37:58.828 INFO clienttrace - src: /127.0.0.1:54546, dest: /127.0.0.1:37927, bytes: 592, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1369393814_1, offset: 0, srvID: 379dd5e5-59c3-42cd-b9d3-67ca5e44560b, blockid: BP-617366194-10.1.0.252-1761586664431:blk_1073741861_1037, duration(ns): 535136
17:37:58.828 INFO DataNode - PacketResponder: BP-617366194-10.1.0.252-1761586664431:blk_1073741861_1037, type=LAST_IN_PIPELINE terminating
17:37:58.829 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam.parts/_temporary/0/_temporary/attempt_202510271737578168487553981124570_0234_r_000000_0/.part-r-00000.bai is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:37:58.830 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam.parts/_temporary/0/_temporary/attempt_202510271737578168487553981124570_0234_r_000000_0 dst=null perm=null proto=rpc
17:37:58.831 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam.parts/_temporary/0/_temporary/attempt_202510271737578168487553981124570_0234_r_000000_0 dst=null perm=null proto=rpc
17:37:58.832 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam.parts/_temporary/0/task_202510271737578168487553981124570_0234_r_000000 dst=null perm=null proto=rpc
17:37:58.832 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=rename src=/user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam.parts/_temporary/0/_temporary/attempt_202510271737578168487553981124570_0234_r_000000_0 dst=/user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam.parts/_temporary/0/task_202510271737578168487553981124570_0234_r_000000 perm=runner:supergroup:rwxr-xr-x proto=rpc
17:37:58.833 INFO FileOutputCommitter - Saved output of task 'attempt_202510271737578168487553981124570_0234_r_000000_0' to hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam.parts/_temporary/0/task_202510271737578168487553981124570_0234_r_000000
17:37:58.833 INFO SparkHadoopMapRedUtil - attempt_202510271737578168487553981124570_0234_r_000000_0: Committed. Elapsed time: 2 ms.
17:37:58.838 INFO Executor - Finished task 0.0 in stage 60.0 (TID 98). 1944 bytes result sent to driver
17:37:58.839 INFO BlockManagerInfo - Removed broadcast_96_piece0 on localhost:37953 in memory (size: 153.6 KiB, free: 1919.5 MiB)
17:37:58.839 INFO TaskSetManager - Finished task 0.0 in stage 60.0 (TID 98) in 858 ms on localhost (executor driver) (1/1)
17:37:58.839 INFO TaskSchedulerImpl - Removed TaskSet 60.0, whose tasks have all completed, from pool
17:37:58.839 INFO DAGScheduler - ResultStage 60 (runJob at SparkHadoopWriter.scala:83) finished in 0.866 s
17:37:58.839 INFO DAGScheduler - Job 44 is finished. Cancelling potential speculative or zombie tasks for this job
17:37:58.839 INFO TaskSchedulerImpl - Killing all running tasks in stage 60: Stage finished
17:37:58.839 INFO DAGScheduler - Job 44 finished: runJob at SparkHadoopWriter.scala:83, took 0.939644 s
17:37:58.840 INFO SparkHadoopWriter - Start to commit write Job job_202510271737578168487553981124570_0234.
17:37:58.840 INFO BlockManagerInfo - Removed broadcast_93_piece0 on localhost:37953 in memory (size: 233.0 B, free: 1919.5 MiB)
17:37:58.840 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=listStatus src=/user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam.parts/_temporary/0 dst=null perm=null proto=rpc
17:37:58.841 INFO BlockManagerInfo - Removed broadcast_97_piece0 on localhost:37953 in memory (size: 54.6 KiB, free: 1919.6 MiB)
17:37:58.841 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam.parts dst=null perm=null proto=rpc
17:37:58.842 INFO BlockManagerInfo - Removed broadcast_94_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.6 MiB)
17:37:58.842 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=listStatus src=/user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam.parts/_temporary/0/task_202510271737578168487553981124570_0234_r_000000 dst=null perm=null proto=rpc
17:37:58.843 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam.parts/.part-r-00000.bai dst=null perm=null proto=rpc
17:37:58.843 INFO BlockManagerInfo - Removed broadcast_99_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.7 MiB)
17:37:58.843 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=rename src=/user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam.parts/_temporary/0/task_202510271737578168487553981124570_0234_r_000000/.part-r-00000.bai dst=/user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam.parts/.part-r-00000.bai perm=runner:supergroup:rw-r--r-- proto=rpc
17:37:58.844 INFO BlockManagerInfo - Removed broadcast_87_piece0 on localhost:37953 in memory (size: 50.3 KiB, free: 1919.7 MiB)
17:37:58.844 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam.parts/.part-r-00000.sbi dst=null perm=null proto=rpc
17:37:58.844 INFO BlockManagerInfo - Removed broadcast_102_piece0 on localhost:37953 in memory (size: 157.6 KiB, free: 1919.9 MiB)
17:37:58.845 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=rename src=/user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam.parts/_temporary/0/task_202510271737578168487553981124570_0234_r_000000/.part-r-00000.sbi dst=/user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam.parts/.part-r-00000.sbi perm=runner:supergroup:rw-r--r-- proto=rpc
17:37:58.846 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam.parts/part-r-00000 dst=null perm=null proto=rpc
17:37:58.848 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=rename src=/user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam.parts/_temporary/0/task_202510271737578168487553981124570_0234_r_000000/part-r-00000 dst=/user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam.parts/part-r-00000 perm=runner:supergroup:rw-r--r-- proto=rpc
17:37:58.849 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=delete src=/user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam.parts/_temporary dst=null perm=null proto=rpc
17:37:58.850 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam.parts/_SUCCESS dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:37:58.851 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam.parts/_SUCCESS is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:37:58.852 INFO audit - allowed=false ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=delete src=/user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam.parts/.spark-staging-234 dst=null perm=null proto=rpc
17:37:58.852 INFO SparkHadoopWriter - Write Job job_202510271737578168487553981124570_0234 committed. Elapsed time: 12 ms.
17:37:58.853 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam.parts/header dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:37:58.853 WARN IntelDeflaterFactory - Intel Deflater not supported, using Java.util.zip.Deflater
17:37:58.854 INFO StateChange - BLOCK* allocate blk_1073741862_1038, replicas=127.0.0.1:37927 for /user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam.parts/header
17:37:58.855 INFO DataNode - Receiving BP-617366194-10.1.0.252-1761586664431:blk_1073741862_1038 src: /127.0.0.1:54552 dest: /127.0.0.1:37927
17:37:58.856 INFO clienttrace - src: /127.0.0.1:54552, dest: /127.0.0.1:37927, bytes: 1180, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1369393814_1, offset: 0, srvID: 379dd5e5-59c3-42cd-b9d3-67ca5e44560b, blockid: BP-617366194-10.1.0.252-1761586664431:blk_1073741862_1038, duration(ns): 514616
17:37:58.856 INFO DataNode - PacketResponder: BP-617366194-10.1.0.252-1761586664431:blk_1073741862_1038, type=LAST_IN_PIPELINE terminating
17:37:58.857 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam.parts/header is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:37:58.858 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam.parts/terminator dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:37:58.859 INFO StateChange - BLOCK* allocate blk_1073741863_1039, replicas=127.0.0.1:37927 for /user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam.parts/terminator
17:37:58.860 INFO DataNode - Receiving BP-617366194-10.1.0.252-1761586664431:blk_1073741863_1039 src: /127.0.0.1:54566 dest: /127.0.0.1:37927
17:37:58.861 INFO clienttrace - src: /127.0.0.1:54566, dest: /127.0.0.1:37927, bytes: 28, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1369393814_1, offset: 0, srvID: 379dd5e5-59c3-42cd-b9d3-67ca5e44560b, blockid: BP-617366194-10.1.0.252-1761586664431:blk_1073741863_1039, duration(ns): 622250
17:37:58.861 INFO DataNode - PacketResponder: BP-617366194-10.1.0.252-1761586664431:blk_1073741863_1039, type=LAST_IN_PIPELINE terminating
17:37:58.862 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam.parts/terminator is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:37:58.863 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=listStatus src=/user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam.parts dst=null perm=null proto=rpc
17:37:58.864 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam.parts/output dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:37:58.864 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam.parts/output is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:37:58.865 INFO HadoopFileSystemWrapper - Concatenating 3 parts to hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam
17:37:58.865 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=concat src=[/user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam.parts/header, /user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam.parts/part-r-00000, /user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam.parts/terminator] dst=/user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam.parts/output perm=runner:supergroup:rw-r--r-- proto=rpc
17:37:58.866 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam dst=null perm=null proto=rpc
17:37:58.866 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=rename src=/user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam.parts/output dst=/user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam perm=runner:supergroup:rw-r--r-- proto=rpc
17:37:58.867 INFO HadoopFileSystemWrapper - Concatenating to hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam done
17:37:58.867 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam dst=null perm=null proto=rpc
17:37:58.867 INFO IndexFileMerger - Merging .sbi files in temp directory hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam.parts/ to hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam.sbi
17:37:58.868 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=listStatus src=/user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam.parts dst=null perm=null proto=rpc
17:37:58.869 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam.sbi dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:37:58.870 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam.parts/.part-r-00000.sbi dst=null perm=null proto=rpc
17:37:58.870 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam.parts/.part-r-00000.sbi dst=null perm=null proto=rpc
17:37:58.872 WARN DFSUtil - Unexpected value for data transfer bytes=208 duration=0
17:37:58.872 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=delete src=/user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam.parts/.part-r-00000.sbi dst=null perm=null proto=rpc
17:37:58.873 INFO StateChange - BLOCK* allocate blk_1073741864_1040, replicas=127.0.0.1:37927 for /user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam.sbi
17:37:58.874 INFO DataNode - Receiving BP-617366194-10.1.0.252-1761586664431:blk_1073741864_1040 src: /127.0.0.1:54568 dest: /127.0.0.1:37927
17:37:58.875 INFO clienttrace - src: /127.0.0.1:54568, dest: /127.0.0.1:37927, bytes: 204, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1369393814_1, offset: 0, srvID: 379dd5e5-59c3-42cd-b9d3-67ca5e44560b, blockid: BP-617366194-10.1.0.252-1761586664431:blk_1073741864_1040, duration(ns): 516384
17:37:58.876 INFO DataNode - PacketResponder: BP-617366194-10.1.0.252-1761586664431:blk_1073741864_1040, type=LAST_IN_PIPELINE terminating
17:37:58.876 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam.sbi is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:37:58.877 INFO IndexFileMerger - Done merging .sbi files
17:37:58.877 INFO IndexFileMerger - Merging .bai files in temp directory hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam.parts/ to hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam.bai
17:37:58.877 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=listStatus src=/user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam.parts dst=null perm=null proto=rpc
17:37:58.878 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam.bai dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:37:58.879 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam.parts/.part-r-00000.bai dst=null perm=null proto=rpc
17:37:58.879 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam.parts/.part-r-00000.bai dst=null perm=null proto=rpc
17:37:58.880 WARN DFSUtil - Unexpected value for data transfer bytes=600 duration=0
17:37:58.881 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=delete src=/user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam.parts/.part-r-00000.bai dst=null perm=null proto=rpc
17:37:58.882 INFO StateChange - BLOCK* allocate blk_1073741865_1041, replicas=127.0.0.1:37927 for /user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam.bai
17:37:58.884 INFO DataNode - Receiving BP-617366194-10.1.0.252-1761586664431:blk_1073741865_1041 src: /127.0.0.1:54570 dest: /127.0.0.1:37927
17:37:58.885 INFO clienttrace - src: /127.0.0.1:54570, dest: /127.0.0.1:37927, bytes: 592, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1369393814_1, offset: 0, srvID: 379dd5e5-59c3-42cd-b9d3-67ca5e44560b, blockid: BP-617366194-10.1.0.252-1761586664431:blk_1073741865_1041, duration(ns): 543384
17:37:58.885 INFO DataNode - PacketResponder: BP-617366194-10.1.0.252-1761586664431:blk_1073741865_1041, type=LAST_IN_PIPELINE terminating
17:37:58.886 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam.bai is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:37:58.886 INFO IndexFileMerger - Done merging .bai files
17:37:58.886 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=delete src=/user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam.parts dst=null perm=null proto=rpc
17:37:58.896 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam.bai dst=null perm=null proto=rpc
17:37:58.904 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam.sbi dst=null perm=null proto=rpc
17:37:58.904 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam.sbi dst=null perm=null proto=rpc
17:37:58.905 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam.sbi dst=null perm=null proto=rpc
17:37:58.906 WARN DFSUtil - Unexpected value for data transfer bytes=208 duration=0
17:37:58.907 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam dst=null perm=null proto=rpc
17:37:58.907 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam dst=null perm=null proto=rpc
17:37:58.908 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam dst=null perm=null proto=rpc
17:37:58.908 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam dst=null perm=null proto=rpc
17:37:58.910 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam.bai dst=null perm=null proto=rpc
17:37:58.910 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam.bai dst=null perm=null proto=rpc
17:37:58.911 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam.bai dst=null perm=null proto=rpc
17:37:58.912 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:37:58.913 WARN DFSUtil - Unexpected value for data transfer bytes=1192 duration=0
17:37:58.914 WARN DFSUtil - Unexpected value for data transfer bytes=600 duration=0
17:37:58.915 WARN DFSUtil - Unexpected value for data transfer bytes=600 duration=0
17:37:58.915 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam.sbi dst=null perm=null proto=rpc
17:37:58.916 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam.sbi dst=null perm=null proto=rpc
17:37:58.916 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam.sbi dst=null perm=null proto=rpc
17:37:58.918 INFO MemoryStore - Block broadcast_104 stored as values in memory (estimated size 312.0 B, free 1919.4 MiB)
17:37:58.918 INFO MemoryStore - Block broadcast_104_piece0 stored as bytes in memory (estimated size 231.0 B, free 1919.4 MiB)
17:37:58.919 INFO BlockManagerInfo - Added broadcast_104_piece0 in memory on localhost:37953 (size: 231.0 B, free: 1919.9 MiB)
17:37:58.919 INFO SparkContext - Created broadcast 104 from broadcast at BamSource.java:104
17:37:58.920 INFO MemoryStore - Block broadcast_105 stored as values in memory (estimated size 297.9 KiB, free 1919.1 MiB)
17:37:58.926 INFO MemoryStore - Block broadcast_105_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1919.1 MiB)
17:37:58.926 INFO BlockManagerInfo - Added broadcast_105_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.8 MiB)
17:37:58.926 INFO SparkContext - Created broadcast 105 from newAPIHadoopFile at PathSplitSource.java:96
17:37:58.937 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam dst=null perm=null proto=rpc
17:37:58.937 INFO FileInputFormat - Total input files to process : 1
17:37:58.937 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam dst=null perm=null proto=rpc
17:37:58.951 INFO SparkContext - Starting job: collect at ReadsSparkSinkUnitTest.java:182
17:37:58.952 INFO DAGScheduler - Got job 45 (collect at ReadsSparkSinkUnitTest.java:182) with 1 output partitions
17:37:58.952 INFO DAGScheduler - Final stage: ResultStage 61 (collect at ReadsSparkSinkUnitTest.java:182)
17:37:58.952 INFO DAGScheduler - Parents of final stage: List()
17:37:58.952 INFO DAGScheduler - Missing parents: List()
17:37:58.952 INFO DAGScheduler - Submitting ResultStage 61 (MapPartitionsRDD[240] at filter at ReadsSparkSource.java:96), which has no missing parents
17:37:58.958 INFO MemoryStore - Block broadcast_106 stored as values in memory (estimated size 148.2 KiB, free 1918.9 MiB)
17:37:58.959 INFO MemoryStore - Block broadcast_106_piece0 stored as bytes in memory (estimated size 54.6 KiB, free 1918.9 MiB)
17:37:58.959 INFO BlockManagerInfo - Added broadcast_106_piece0 in memory on localhost:37953 (size: 54.6 KiB, free: 1919.8 MiB)
17:37:58.960 INFO SparkContext - Created broadcast 106 from broadcast at DAGScheduler.scala:1580
17:37:58.960 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 61 (MapPartitionsRDD[240] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:37:58.960 INFO TaskSchedulerImpl - Adding task set 61.0 with 1 tasks resource profile 0
17:37:58.961 INFO TaskSetManager - Starting task 0.0 in stage 61.0 (TID 99) (localhost, executor driver, partition 0, ANY, 7852 bytes)
17:37:58.961 INFO Executor - Running task 0.0 in stage 61.0 (TID 99)
17:37:58.974 INFO NewHadoopRDD - Input split: hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam:0+237078
17:37:58.976 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam dst=null perm=null proto=rpc
17:37:58.976 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam dst=null perm=null proto=rpc
17:37:58.978 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam.bai dst=null perm=null proto=rpc
17:37:58.978 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam.bai dst=null perm=null proto=rpc
17:37:58.979 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam.bai dst=null perm=null proto=rpc
17:37:58.980 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:37:58.981 WARN DFSUtil - Unexpected value for data transfer bytes=1192 duration=0
17:37:58.983 WARN DFSUtil - Unexpected value for data transfer bytes=600 duration=0
17:37:58.984 WARN DFSUtil - Unexpected value for data transfer bytes=237714 duration=0
17:37:58.985 WARN DFSUtil - Unexpected value for data transfer bytes=32 duration=0
17:37:58.988 INFO Executor - Finished task 0.0 in stage 61.0 (TID 99). 749470 bytes result sent to driver
17:37:58.989 INFO TaskSetManager - Finished task 0.0 in stage 61.0 (TID 99) in 28 ms on localhost (executor driver) (1/1)
17:37:58.989 INFO TaskSchedulerImpl - Removed TaskSet 61.0, whose tasks have all completed, from pool
17:37:58.990 INFO DAGScheduler - ResultStage 61 (collect at ReadsSparkSinkUnitTest.java:182) finished in 0.037 s
17:37:58.990 INFO DAGScheduler - Job 45 is finished. Cancelling potential speculative or zombie tasks for this job
17:37:58.990 INFO TaskSchedulerImpl - Killing all running tasks in stage 61: Stage finished
17:37:58.990 INFO DAGScheduler - Job 45 finished: collect at ReadsSparkSinkUnitTest.java:182, took 0.038459 s
17:37:59.005 INFO SparkContext - Starting job: count at ReadsSparkSinkUnitTest.java:185
17:37:59.005 INFO DAGScheduler - Got job 46 (count at ReadsSparkSinkUnitTest.java:185) with 1 output partitions
17:37:59.005 INFO DAGScheduler - Final stage: ResultStage 62 (count at ReadsSparkSinkUnitTest.java:185)
17:37:59.005 INFO DAGScheduler - Parents of final stage: List()
17:37:59.005 INFO DAGScheduler - Missing parents: List()
17:37:59.006 INFO DAGScheduler - Submitting ResultStage 62 (MapPartitionsRDD[222] at filter at ReadsSparkSource.java:96), which has no missing parents
17:37:59.021 INFO MemoryStore - Block broadcast_107 stored as values in memory (estimated size 426.1 KiB, free 1918.5 MiB)
17:37:59.022 INFO MemoryStore - Block broadcast_107_piece0 stored as bytes in memory (estimated size 153.6 KiB, free 1918.3 MiB)
17:37:59.023 INFO BlockManagerInfo - Added broadcast_107_piece0 in memory on localhost:37953 (size: 153.6 KiB, free: 1919.6 MiB)
17:37:59.023 INFO SparkContext - Created broadcast 107 from broadcast at DAGScheduler.scala:1580
17:37:59.023 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 62 (MapPartitionsRDD[222] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:37:59.023 INFO TaskSchedulerImpl - Adding task set 62.0 with 1 tasks resource profile 0
17:37:59.024 INFO TaskSetManager - Starting task 0.0 in stage 62.0 (TID 100) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7893 bytes)
17:37:59.024 INFO Executor - Running task 0.0 in stage 62.0 (TID 100)
17:37:59.059 INFO NewHadoopRDD - Input split: file:/home/runner/work/gatk/gatk/src/test/resources/org/broadinstitute/hellbender/tools/BQSR/CEUTrio.HiSeq.WGS.b37.ch20.1m-1m1k.NA12878.bam:0+211123
17:37:59.060 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:37:59.060 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:37:59.067 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:37:59.070 INFO Executor - Finished task 0.0 in stage 62.0 (TID 100). 989 bytes result sent to driver
17:37:59.071 INFO TaskSetManager - Finished task 0.0 in stage 62.0 (TID 100) in 47 ms on localhost (executor driver) (1/1)
17:37:59.071 INFO TaskSchedulerImpl - Removed TaskSet 62.0, whose tasks have all completed, from pool
17:37:59.071 INFO DAGScheduler - ResultStage 62 (count at ReadsSparkSinkUnitTest.java:185) finished in 0.065 s
17:37:59.071 INFO DAGScheduler - Job 46 is finished. Cancelling potential speculative or zombie tasks for this job
17:37:59.071 INFO TaskSchedulerImpl - Killing all running tasks in stage 62: Stage finished
17:37:59.071 INFO DAGScheduler - Job 46 finished: count at ReadsSparkSinkUnitTest.java:185, took 0.066553 s
17:37:59.075 INFO SparkContext - Starting job: count at ReadsSparkSinkUnitTest.java:185
17:37:59.075 INFO DAGScheduler - Got job 47 (count at ReadsSparkSinkUnitTest.java:185) with 1 output partitions
17:37:59.075 INFO DAGScheduler - Final stage: ResultStage 63 (count at ReadsSparkSinkUnitTest.java:185)
17:37:59.075 INFO DAGScheduler - Parents of final stage: List()
17:37:59.075 INFO DAGScheduler - Missing parents: List()
17:37:59.075 INFO DAGScheduler - Submitting ResultStage 63 (MapPartitionsRDD[240] at filter at ReadsSparkSource.java:96), which has no missing parents
17:37:59.081 INFO MemoryStore - Block broadcast_108 stored as values in memory (estimated size 148.1 KiB, free 1918.2 MiB)
17:37:59.082 INFO MemoryStore - Block broadcast_108_piece0 stored as bytes in memory (estimated size 54.6 KiB, free 1918.1 MiB)
17:37:59.082 INFO BlockManagerInfo - Added broadcast_108_piece0 in memory on localhost:37953 (size: 54.6 KiB, free: 1919.6 MiB)
17:37:59.083 INFO SparkContext - Created broadcast 108 from broadcast at DAGScheduler.scala:1580
17:37:59.083 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 63 (MapPartitionsRDD[240] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:37:59.083 INFO TaskSchedulerImpl - Adding task set 63.0 with 1 tasks resource profile 0
17:37:59.084 INFO TaskSetManager - Starting task 0.0 in stage 63.0 (TID 101) (localhost, executor driver, partition 0, ANY, 7852 bytes)
17:37:59.084 INFO Executor - Running task 0.0 in stage 63.0 (TID 101)
17:37:59.097 INFO NewHadoopRDD - Input split: hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam:0+237078
17:37:59.098 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam dst=null perm=null proto=rpc
17:37:59.099 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam dst=null perm=null proto=rpc
17:37:59.100 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam.bai dst=null perm=null proto=rpc
17:37:59.101 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam.bai dst=null perm=null proto=rpc
17:37:59.101 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest3_a985221a-49b6-401d-9e34-004f8f2100b8.bam.bai dst=null perm=null proto=rpc
17:37:59.103 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:37:59.103 WARN DFSUtil - Unexpected value for data transfer bytes=1192 duration=0
17:37:59.105 WARN DFSUtil - Unexpected value for data transfer bytes=600 duration=0
17:37:59.105 WARN DFSUtil - Unexpected value for data transfer bytes=600 duration=0
17:37:59.107 WARN DFSUtil - Unexpected value for data transfer bytes=237714 duration=0
17:37:59.107 WARN DFSUtil - Unexpected value for data transfer bytes=32 duration=0
17:37:59.109 INFO Executor - Finished task 0.0 in stage 63.0 (TID 101). 989 bytes result sent to driver
17:37:59.110 INFO TaskSetManager - Finished task 0.0 in stage 63.0 (TID 101) in 27 ms on localhost (executor driver) (1/1)
17:37:59.110 INFO TaskSchedulerImpl - Removed TaskSet 63.0, whose tasks have all completed, from pool
17:37:59.110 INFO DAGScheduler - ResultStage 63 (count at ReadsSparkSinkUnitTest.java:185) finished in 0.034 s
17:37:59.110 INFO DAGScheduler - Job 47 is finished. Cancelling potential speculative or zombie tasks for this job
17:37:59.110 INFO TaskSchedulerImpl - Killing all running tasks in stage 63: Stage finished
17:37:59.110 INFO DAGScheduler - Job 47 finished: count at ReadsSparkSinkUnitTest.java:185, took 0.035430 s
17:37:59.117 INFO MemoryStore - Block broadcast_109 stored as values in memory (estimated size 576.0 B, free 1918.1 MiB)
17:37:59.120 INFO MemoryStore - Block broadcast_109_piece0 stored as bytes in memory (estimated size 228.0 B, free 1918.1 MiB)
17:37:59.120 INFO BlockManagerInfo - Added broadcast_109_piece0 in memory on localhost:37953 (size: 228.0 B, free: 1919.6 MiB)
17:37:59.120 INFO SparkContext - Created broadcast 109 from broadcast at CramSource.java:114
17:37:59.121 INFO MemoryStore - Block broadcast_110 stored as values in memory (estimated size 297.9 KiB, free 1917.8 MiB)
17:37:59.127 INFO MemoryStore - Block broadcast_110_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1917.8 MiB)
17:37:59.127 INFO BlockManagerInfo - Added broadcast_110_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.5 MiB)
17:37:59.128 INFO SparkContext - Created broadcast 110 from newAPIHadoopFile at PathSplitSource.java:96
17:37:59.146 INFO MemoryStore - Block broadcast_111 stored as values in memory (estimated size 576.0 B, free 1917.8 MiB)
17:37:59.147 INFO MemoryStore - Block broadcast_111_piece0 stored as bytes in memory (estimated size 228.0 B, free 1917.8 MiB)
17:37:59.147 INFO BlockManagerInfo - Added broadcast_111_piece0 in memory on localhost:37953 (size: 228.0 B, free: 1919.5 MiB)
17:37:59.147 INFO SparkContext - Created broadcast 111 from broadcast at CramSource.java:114
17:37:59.149 INFO MemoryStore - Block broadcast_112 stored as values in memory (estimated size 297.9 KiB, free 1917.5 MiB)
17:37:59.156 INFO BlockManagerInfo - Removed broadcast_106_piece0 on localhost:37953 in memory (size: 54.6 KiB, free: 1919.6 MiB)
17:37:59.158 INFO BlockManagerInfo - Removed broadcast_105_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.6 MiB)
17:37:59.159 INFO BlockManagerInfo - Removed broadcast_103_piece0 on localhost:37953 in memory (size: 58.5 KiB, free: 1919.7 MiB)
17:37:59.159 INFO BlockManagerInfo - Removed broadcast_107_piece0 on localhost:37953 in memory (size: 153.6 KiB, free: 1919.8 MiB)
17:37:59.159 INFO MemoryStore - Block broadcast_112_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1918.7 MiB)
17:37:59.160 INFO BlockManagerInfo - Added broadcast_112_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.8 MiB)
17:37:59.160 INFO SparkContext - Created broadcast 112 from newAPIHadoopFile at PathSplitSource.java:96
17:37:59.160 INFO BlockManagerInfo - Removed broadcast_104_piece0 on localhost:37953 in memory (size: 231.0 B, free: 1919.8 MiB)
17:37:59.161 INFO BlockManagerInfo - Removed broadcast_98_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.8 MiB)
17:37:59.161 INFO BlockManagerInfo - Removed broadcast_108_piece0 on localhost:37953 in memory (size: 54.6 KiB, free: 1919.9 MiB)
17:37:59.162 INFO BlockManagerInfo - Removed broadcast_101_piece0 on localhost:37953 in memory (size: 1890.0 B, free: 1919.9 MiB)
17:37:59.163 INFO BlockManagerInfo - Removed broadcast_100_piece0 on localhost:37953 in memory (size: 1890.0 B, free: 1919.9 MiB)
17:37:59.175 INFO FileInputFormat - Total input files to process : 1
17:37:59.177 INFO MemoryStore - Block broadcast_113 stored as values in memory (estimated size 6.0 KiB, free 1919.3 MiB)
17:37:59.177 INFO MemoryStore - Block broadcast_113_piece0 stored as bytes in memory (estimated size 1473.0 B, free 1919.3 MiB)
17:37:59.177 INFO BlockManagerInfo - Added broadcast_113_piece0 in memory on localhost:37953 (size: 1473.0 B, free: 1919.9 MiB)
17:37:59.178 INFO SparkContext - Created broadcast 113 from broadcast at ReadsSparkSink.java:133
17:37:59.179 INFO MemoryStore - Block broadcast_114 stored as values in memory (estimated size 6.2 KiB, free 1919.3 MiB)
17:37:59.179 INFO MemoryStore - Block broadcast_114_piece0 stored as bytes in memory (estimated size 1473.0 B, free 1919.3 MiB)
17:37:59.179 INFO BlockManagerInfo - Added broadcast_114_piece0 in memory on localhost:37953 (size: 1473.0 B, free: 1919.9 MiB)
17:37:59.180 INFO SparkContext - Created broadcast 114 from broadcast at CramSink.java:76
17:37:59.185 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest5_f0550c4d-35fe-4b14-841a-b6a8c2c2b6b9.cram.parts dst=null perm=null proto=rpc
17:37:59.186 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:37:59.186 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:37:59.186 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:37:59.187 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=mkdirs src=/user/runner/ReadsSparkSinkUnitTest5_f0550c4d-35fe-4b14-841a-b6a8c2c2b6b9.cram.parts/_temporary/0 dst=null perm=runner:supergroup:rwxr-xr-x proto=rpc
17:37:59.193 INFO SparkContext - Starting job: runJob at SparkHadoopWriter.scala:83
17:37:59.193 INFO DAGScheduler - Registering RDD 252 (mapToPair at SparkUtils.java:161) as input to shuffle 14
17:37:59.194 INFO DAGScheduler - Got job 48 (runJob at SparkHadoopWriter.scala:83) with 1 output partitions
17:37:59.194 INFO DAGScheduler - Final stage: ResultStage 65 (runJob at SparkHadoopWriter.scala:83)
17:37:59.194 INFO DAGScheduler - Parents of final stage: List(ShuffleMapStage 64)
17:37:59.194 INFO DAGScheduler - Missing parents: List(ShuffleMapStage 64)
17:37:59.194 INFO DAGScheduler - Submitting ShuffleMapStage 64 (MapPartitionsRDD[252] at mapToPair at SparkUtils.java:161), which has no missing parents
17:37:59.205 INFO MemoryStore - Block broadcast_115 stored as values in memory (estimated size 292.8 KiB, free 1919.0 MiB)
17:37:59.206 INFO MemoryStore - Block broadcast_115_piece0 stored as bytes in memory (estimated size 107.3 KiB, free 1918.9 MiB)
17:37:59.206 INFO BlockManagerInfo - Added broadcast_115_piece0 in memory on localhost:37953 (size: 107.3 KiB, free: 1919.8 MiB)
17:37:59.206 INFO SparkContext - Created broadcast 115 from broadcast at DAGScheduler.scala:1580
17:37:59.207 INFO DAGScheduler - Submitting 1 missing tasks from ShuffleMapStage 64 (MapPartitionsRDD[252] at mapToPair at SparkUtils.java:161) (first 15 tasks are for partitions Vector(0))
17:37:59.207 INFO TaskSchedulerImpl - Adding task set 64.0 with 1 tasks resource profile 0
17:37:59.208 INFO TaskSetManager - Starting task 0.0 in stage 64.0 (TID 102) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7869 bytes)
17:37:59.208 INFO Executor - Running task 0.0 in stage 64.0 (TID 102)
17:37:59.233 INFO NewHadoopRDD - Input split: file:/home/runner/work/gatk/gatk/src/test/resources/org/broadinstitute/hellbender/tools/BQSR/NA12878.chr17_69k_70k.dictFix.cram:0+50619
17:37:59.256 INFO Executor - Finished task 0.0 in stage 64.0 (TID 102). 1148 bytes result sent to driver
17:37:59.257 INFO TaskSetManager - Finished task 0.0 in stage 64.0 (TID 102) in 50 ms on localhost (executor driver) (1/1)
17:37:59.257 INFO TaskSchedulerImpl - Removed TaskSet 64.0, whose tasks have all completed, from pool
17:37:59.257 INFO DAGScheduler - ShuffleMapStage 64 (mapToPair at SparkUtils.java:161) finished in 0.063 s
17:37:59.257 INFO DAGScheduler - looking for newly runnable stages
17:37:59.257 INFO DAGScheduler - running: HashSet()
17:37:59.257 INFO DAGScheduler - waiting: HashSet(ResultStage 65)
17:37:59.257 INFO DAGScheduler - failed: HashSet()
17:37:59.257 INFO DAGScheduler - Submitting ResultStage 65 (MapPartitionsRDD[257] at mapToPair at CramSink.java:89), which has no missing parents
17:37:59.264 INFO MemoryStore - Block broadcast_116 stored as values in memory (estimated size 153.3 KiB, free 1918.8 MiB)
17:37:59.265 INFO MemoryStore - Block broadcast_116_piece0 stored as bytes in memory (estimated size 58.1 KiB, free 1918.7 MiB)
17:37:59.265 INFO BlockManagerInfo - Added broadcast_116_piece0 in memory on localhost:37953 (size: 58.1 KiB, free: 1919.7 MiB)
17:37:59.265 INFO SparkContext - Created broadcast 116 from broadcast at DAGScheduler.scala:1580
17:37:59.265 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 65 (MapPartitionsRDD[257] at mapToPair at CramSink.java:89) (first 15 tasks are for partitions Vector(0))
17:37:59.265 INFO TaskSchedulerImpl - Adding task set 65.0 with 1 tasks resource profile 0
17:37:59.266 INFO TaskSetManager - Starting task 0.0 in stage 65.0 (TID 103) (localhost, executor driver, partition 0, NODE_LOCAL, 7513 bytes)
17:37:59.267 INFO Executor - Running task 0.0 in stage 65.0 (TID 103)
17:37:59.273 INFO ShuffleBlockFetcherIterator - Getting 1 (82.3 KiB) non-empty blocks including 1 (82.3 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
17:37:59.274 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
17:37:59.282 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:37:59.282 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:37:59.282 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:37:59.282 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:37:59.282 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:37:59.282 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:37:59.286 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest5_f0550c4d-35fe-4b14-841a-b6a8c2c2b6b9.cram.parts/_temporary/0/_temporary/attempt_202510271737594293920164303648783_0257_r_000000_0/part-r-00000 dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:37:59.379 INFO StateChange - BLOCK* allocate blk_1073741866_1042, replicas=127.0.0.1:37927 for /user/runner/ReadsSparkSinkUnitTest5_f0550c4d-35fe-4b14-841a-b6a8c2c2b6b9.cram.parts/_temporary/0/_temporary/attempt_202510271737594293920164303648783_0257_r_000000_0/part-r-00000
17:37:59.380 INFO DataNode - Receiving BP-617366194-10.1.0.252-1761586664431:blk_1073741866_1042 src: /127.0.0.1:54598 dest: /127.0.0.1:37927
17:37:59.381 INFO clienttrace - src: /127.0.0.1:54598, dest: /127.0.0.1:37927, bytes: 42659, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1369393814_1, offset: 0, srvID: 379dd5e5-59c3-42cd-b9d3-67ca5e44560b, blockid: BP-617366194-10.1.0.252-1761586664431:blk_1073741866_1042, duration(ns): 607898
17:37:59.381 INFO DataNode - PacketResponder: BP-617366194-10.1.0.252-1761586664431:blk_1073741866_1042, type=LAST_IN_PIPELINE terminating
17:37:59.382 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest5_f0550c4d-35fe-4b14-841a-b6a8c2c2b6b9.cram.parts/_temporary/0/_temporary/attempt_202510271737594293920164303648783_0257_r_000000_0/part-r-00000 is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:37:59.383 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest5_f0550c4d-35fe-4b14-841a-b6a8c2c2b6b9.cram.parts/_temporary/0/_temporary/attempt_202510271737594293920164303648783_0257_r_000000_0 dst=null perm=null proto=rpc
17:37:59.384 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest5_f0550c4d-35fe-4b14-841a-b6a8c2c2b6b9.cram.parts/_temporary/0/_temporary/attempt_202510271737594293920164303648783_0257_r_000000_0 dst=null perm=null proto=rpc
17:37:59.385 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest5_f0550c4d-35fe-4b14-841a-b6a8c2c2b6b9.cram.parts/_temporary/0/task_202510271737594293920164303648783_0257_r_000000 dst=null perm=null proto=rpc
17:37:59.385 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=rename src=/user/runner/ReadsSparkSinkUnitTest5_f0550c4d-35fe-4b14-841a-b6a8c2c2b6b9.cram.parts/_temporary/0/_temporary/attempt_202510271737594293920164303648783_0257_r_000000_0 dst=/user/runner/ReadsSparkSinkUnitTest5_f0550c4d-35fe-4b14-841a-b6a8c2c2b6b9.cram.parts/_temporary/0/task_202510271737594293920164303648783_0257_r_000000 perm=runner:supergroup:rwxr-xr-x proto=rpc
17:37:59.386 INFO FileOutputCommitter - Saved output of task 'attempt_202510271737594293920164303648783_0257_r_000000_0' to hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest5_f0550c4d-35fe-4b14-841a-b6a8c2c2b6b9.cram.parts/_temporary/0/task_202510271737594293920164303648783_0257_r_000000
17:37:59.386 INFO SparkHadoopMapRedUtil - attempt_202510271737594293920164303648783_0257_r_000000_0: Committed. Elapsed time: 1 ms.
17:37:59.386 INFO Executor - Finished task 0.0 in stage 65.0 (TID 103). 1858 bytes result sent to driver
17:37:59.387 INFO TaskSetManager - Finished task 0.0 in stage 65.0 (TID 103) in 121 ms on localhost (executor driver) (1/1)
17:37:59.387 INFO TaskSchedulerImpl - Removed TaskSet 65.0, whose tasks have all completed, from pool
17:37:59.387 INFO DAGScheduler - ResultStage 65 (runJob at SparkHadoopWriter.scala:83) finished in 0.129 s
17:37:59.387 INFO DAGScheduler - Job 48 is finished. Cancelling potential speculative or zombie tasks for this job
17:37:59.387 INFO TaskSchedulerImpl - Killing all running tasks in stage 65: Stage finished
17:37:59.388 INFO DAGScheduler - Job 48 finished: runJob at SparkHadoopWriter.scala:83, took 0.195025 s
17:37:59.388 INFO SparkHadoopWriter - Start to commit write Job job_202510271737594293920164303648783_0257.
17:37:59.389 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=listStatus src=/user/runner/ReadsSparkSinkUnitTest5_f0550c4d-35fe-4b14-841a-b6a8c2c2b6b9.cram.parts/_temporary/0 dst=null perm=null proto=rpc
17:37:59.389 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest5_f0550c4d-35fe-4b14-841a-b6a8c2c2b6b9.cram.parts dst=null perm=null proto=rpc
17:37:59.390 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=listStatus src=/user/runner/ReadsSparkSinkUnitTest5_f0550c4d-35fe-4b14-841a-b6a8c2c2b6b9.cram.parts/_temporary/0/task_202510271737594293920164303648783_0257_r_000000 dst=null perm=null proto=rpc
17:37:59.390 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest5_f0550c4d-35fe-4b14-841a-b6a8c2c2b6b9.cram.parts/part-r-00000 dst=null perm=null proto=rpc
17:37:59.394 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=rename src=/user/runner/ReadsSparkSinkUnitTest5_f0550c4d-35fe-4b14-841a-b6a8c2c2b6b9.cram.parts/_temporary/0/task_202510271737594293920164303648783_0257_r_000000/part-r-00000 dst=/user/runner/ReadsSparkSinkUnitTest5_f0550c4d-35fe-4b14-841a-b6a8c2c2b6b9.cram.parts/part-r-00000 perm=runner:supergroup:rw-r--r-- proto=rpc
17:37:59.395 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=delete src=/user/runner/ReadsSparkSinkUnitTest5_f0550c4d-35fe-4b14-841a-b6a8c2c2b6b9.cram.parts/_temporary dst=null perm=null proto=rpc
17:37:59.395 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest5_f0550c4d-35fe-4b14-841a-b6a8c2c2b6b9.cram.parts/_SUCCESS dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:37:59.397 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest5_f0550c4d-35fe-4b14-841a-b6a8c2c2b6b9.cram.parts/_SUCCESS is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:37:59.397 INFO audit - allowed=false ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=delete src=/user/runner/ReadsSparkSinkUnitTest5_f0550c4d-35fe-4b14-841a-b6a8c2c2b6b9.cram.parts/.spark-staging-257 dst=null perm=null proto=rpc
17:37:59.398 INFO SparkHadoopWriter - Write Job job_202510271737594293920164303648783_0257 committed. Elapsed time: 9 ms.
17:37:59.398 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest5_f0550c4d-35fe-4b14-841a-b6a8c2c2b6b9.cram.parts/header dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:37:59.401 INFO StateChange - BLOCK* allocate blk_1073741867_1043, replicas=127.0.0.1:37927 for /user/runner/ReadsSparkSinkUnitTest5_f0550c4d-35fe-4b14-841a-b6a8c2c2b6b9.cram.parts/header
17:37:59.402 INFO DataNode - Receiving BP-617366194-10.1.0.252-1761586664431:blk_1073741867_1043 src: /127.0.0.1:54604 dest: /127.0.0.1:37927
17:37:59.403 INFO clienttrace - src: /127.0.0.1:54604, dest: /127.0.0.1:37927, bytes: 1016, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1369393814_1, offset: 0, srvID: 379dd5e5-59c3-42cd-b9d3-67ca5e44560b, blockid: BP-617366194-10.1.0.252-1761586664431:blk_1073741867_1043, duration(ns): 455854
17:37:59.403 INFO DataNode - PacketResponder: BP-617366194-10.1.0.252-1761586664431:blk_1073741867_1043, type=LAST_IN_PIPELINE terminating
17:37:59.404 INFO FSNamesystem - BLOCK* blk_1073741867_1043 is COMMITTED but not COMPLETE(numNodes= 0 < minimum = 1) in file /user/runner/ReadsSparkSinkUnitTest5_f0550c4d-35fe-4b14-841a-b6a8c2c2b6b9.cram.parts/header
17:37:59.805 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest5_f0550c4d-35fe-4b14-841a-b6a8c2c2b6b9.cram.parts/header is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:37:59.806 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest5_f0550c4d-35fe-4b14-841a-b6a8c2c2b6b9.cram.parts/terminator dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:37:59.808 INFO StateChange - BLOCK* allocate blk_1073741868_1044, replicas=127.0.0.1:37927 for /user/runner/ReadsSparkSinkUnitTest5_f0550c4d-35fe-4b14-841a-b6a8c2c2b6b9.cram.parts/terminator
17:37:59.809 INFO DataNode - Receiving BP-617366194-10.1.0.252-1761586664431:blk_1073741868_1044 src: /127.0.0.1:54612 dest: /127.0.0.1:37927
17:37:59.810 INFO clienttrace - src: /127.0.0.1:54612, dest: /127.0.0.1:37927, bytes: 38, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1369393814_1, offset: 0, srvID: 379dd5e5-59c3-42cd-b9d3-67ca5e44560b, blockid: BP-617366194-10.1.0.252-1761586664431:blk_1073741868_1044, duration(ns): 836513
17:37:59.810 INFO DataNode - PacketResponder: BP-617366194-10.1.0.252-1761586664431:blk_1073741868_1044, type=LAST_IN_PIPELINE terminating
17:37:59.811 INFO FSNamesystem - BLOCK* blk_1073741868_1044 is COMMITTED but not COMPLETE(numNodes= 0 < minimum = 1) in file /user/runner/ReadsSparkSinkUnitTest5_f0550c4d-35fe-4b14-841a-b6a8c2c2b6b9.cram.parts/terminator
17:38:00.212 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest5_f0550c4d-35fe-4b14-841a-b6a8c2c2b6b9.cram.parts/terminator is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:38:00.213 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=listStatus src=/user/runner/ReadsSparkSinkUnitTest5_f0550c4d-35fe-4b14-841a-b6a8c2c2b6b9.cram.parts dst=null perm=null proto=rpc
17:38:00.215 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest5_f0550c4d-35fe-4b14-841a-b6a8c2c2b6b9.cram.parts/output dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:00.216 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest5_f0550c4d-35fe-4b14-841a-b6a8c2c2b6b9.cram.parts/output is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:38:00.216 INFO HadoopFileSystemWrapper - Concatenating 3 parts to hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest5_f0550c4d-35fe-4b14-841a-b6a8c2c2b6b9.cram
17:38:00.216 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=concat src=[/user/runner/ReadsSparkSinkUnitTest5_f0550c4d-35fe-4b14-841a-b6a8c2c2b6b9.cram.parts/header, /user/runner/ReadsSparkSinkUnitTest5_f0550c4d-35fe-4b14-841a-b6a8c2c2b6b9.cram.parts/part-r-00000, /user/runner/ReadsSparkSinkUnitTest5_f0550c4d-35fe-4b14-841a-b6a8c2c2b6b9.cram.parts/terminator] dst=/user/runner/ReadsSparkSinkUnitTest5_f0550c4d-35fe-4b14-841a-b6a8c2c2b6b9.cram.parts/output perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:00.217 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest5_f0550c4d-35fe-4b14-841a-b6a8c2c2b6b9.cram dst=null perm=null proto=rpc
17:38:00.218 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=rename src=/user/runner/ReadsSparkSinkUnitTest5_f0550c4d-35fe-4b14-841a-b6a8c2c2b6b9.cram.parts/output dst=/user/runner/ReadsSparkSinkUnitTest5_f0550c4d-35fe-4b14-841a-b6a8c2c2b6b9.cram perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:00.218 INFO HadoopFileSystemWrapper - Concatenating to hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest5_f0550c4d-35fe-4b14-841a-b6a8c2c2b6b9.cram done
17:38:00.218 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=delete src=/user/runner/ReadsSparkSinkUnitTest5_f0550c4d-35fe-4b14-841a-b6a8c2c2b6b9.cram.parts dst=null perm=null proto=rpc
17:38:00.219 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest5_f0550c4d-35fe-4b14-841a-b6a8c2c2b6b9.cram dst=null perm=null proto=rpc
17:38:00.219 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest5_f0550c4d-35fe-4b14-841a-b6a8c2c2b6b9.cram dst=null perm=null proto=rpc
17:38:00.220 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest5_f0550c4d-35fe-4b14-841a-b6a8c2c2b6b9.cram dst=null perm=null proto=rpc
17:38:00.220 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest5_f0550c4d-35fe-4b14-841a-b6a8c2c2b6b9.cram dst=null perm=null proto=rpc
17:38:00.221 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest5_f0550c4d-35fe-4b14-841a-b6a8c2c2b6b9.cram.crai dst=null perm=null proto=rpc
17:38:00.222 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest5_f0550c4d-35fe-4b14-841a-b6a8c2c2b6b9.crai dst=null perm=null proto=rpc
17:38:00.224 WARN DFSUtil - Unexpected value for data transfer bytes=1024 duration=0
17:38:00.225 WARN DFSUtil - Unexpected value for data transfer bytes=42995 duration=0
17:38:00.225 WARN DFSUtil - Unexpected value for data transfer bytes=42 duration=0
17:38:00.226 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest5_f0550c4d-35fe-4b14-841a-b6a8c2c2b6b9.cram dst=null perm=null proto=rpc
17:38:00.227 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest5_f0550c4d-35fe-4b14-841a-b6a8c2c2b6b9.cram dst=null perm=null proto=rpc
17:38:00.227 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest5_f0550c4d-35fe-4b14-841a-b6a8c2c2b6b9.cram.crai dst=null perm=null proto=rpc
17:38:00.228 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest5_f0550c4d-35fe-4b14-841a-b6a8c2c2b6b9.crai dst=null perm=null proto=rpc
17:38:00.228 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest5_f0550c4d-35fe-4b14-841a-b6a8c2c2b6b9.cram dst=null perm=null proto=rpc
17:38:00.229 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest5_f0550c4d-35fe-4b14-841a-b6a8c2c2b6b9.cram dst=null perm=null proto=rpc
17:38:00.230 WARN DFSUtil - Unexpected value for data transfer bytes=1024 duration=0
17:38:00.230 WARN DFSUtil - Unexpected value for data transfer bytes=42995 duration=0
17:38:00.231 WARN DFSUtil - Unexpected value for data transfer bytes=42 duration=0
17:38:00.231 INFO MemoryStore - Block broadcast_117 stored as values in memory (estimated size 528.0 B, free 1918.7 MiB)
17:38:00.232 INFO MemoryStore - Block broadcast_117_piece0 stored as bytes in memory (estimated size 187.0 B, free 1918.7 MiB)
17:38:00.232 INFO BlockManagerInfo - Added broadcast_117_piece0 in memory on localhost:37953 (size: 187.0 B, free: 1919.7 MiB)
17:38:00.232 INFO SparkContext - Created broadcast 117 from broadcast at CramSource.java:114
17:38:00.233 INFO MemoryStore - Block broadcast_118 stored as values in memory (estimated size 297.9 KiB, free 1918.4 MiB)
17:38:00.239 INFO MemoryStore - Block broadcast_118_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1918.4 MiB)
17:38:00.239 INFO BlockManagerInfo - Added broadcast_118_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.7 MiB)
17:38:00.239 INFO SparkContext - Created broadcast 118 from newAPIHadoopFile at PathSplitSource.java:96
17:38:00.254 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest5_f0550c4d-35fe-4b14-841a-b6a8c2c2b6b9.cram dst=null perm=null proto=rpc
17:38:00.254 INFO FileInputFormat - Total input files to process : 1
17:38:00.254 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest5_f0550c4d-35fe-4b14-841a-b6a8c2c2b6b9.cram dst=null perm=null proto=rpc
17:38:00.277 INFO SparkContext - Starting job: collect at ReadsSparkSinkUnitTest.java:182
17:38:00.278 INFO DAGScheduler - Got job 49 (collect at ReadsSparkSinkUnitTest.java:182) with 1 output partitions
17:38:00.278 INFO DAGScheduler - Final stage: ResultStage 66 (collect at ReadsSparkSinkUnitTest.java:182)
17:38:00.278 INFO DAGScheduler - Parents of final stage: List()
17:38:00.278 INFO DAGScheduler - Missing parents: List()
17:38:00.278 INFO DAGScheduler - Submitting ResultStage 66 (MapPartitionsRDD[263] at filter at ReadsSparkSource.java:96), which has no missing parents
17:38:00.289 INFO MemoryStore - Block broadcast_119 stored as values in memory (estimated size 286.8 KiB, free 1918.1 MiB)
17:38:00.290 INFO MemoryStore - Block broadcast_119_piece0 stored as bytes in memory (estimated size 103.6 KiB, free 1918.0 MiB)
17:38:00.290 INFO BlockManagerInfo - Added broadcast_119_piece0 in memory on localhost:37953 (size: 103.6 KiB, free: 1919.6 MiB)
17:38:00.290 INFO SparkContext - Created broadcast 119 from broadcast at DAGScheduler.scala:1580
17:38:00.290 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 66 (MapPartitionsRDD[263] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:38:00.290 INFO TaskSchedulerImpl - Adding task set 66.0 with 1 tasks resource profile 0
17:38:00.291 INFO TaskSetManager - Starting task 0.0 in stage 66.0 (TID 104) (localhost, executor driver, partition 0, ANY, 7853 bytes)
17:38:00.291 INFO Executor - Running task 0.0 in stage 66.0 (TID 104)
17:38:00.314 INFO NewHadoopRDD - Input split: hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest5_f0550c4d-35fe-4b14-841a-b6a8c2c2b6b9.cram:0+43713
17:38:00.316 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest5_f0550c4d-35fe-4b14-841a-b6a8c2c2b6b9.cram dst=null perm=null proto=rpc
17:38:00.316 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest5_f0550c4d-35fe-4b14-841a-b6a8c2c2b6b9.cram dst=null perm=null proto=rpc
17:38:00.317 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest5_f0550c4d-35fe-4b14-841a-b6a8c2c2b6b9.cram.crai dst=null perm=null proto=rpc
17:38:00.318 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest5_f0550c4d-35fe-4b14-841a-b6a8c2c2b6b9.crai dst=null perm=null proto=rpc
17:38:00.320 WARN DFSUtil - Unexpected value for data transfer bytes=1024 duration=0
17:38:00.321 WARN DFSUtil - Unexpected value for data transfer bytes=42995 duration=0
17:38:00.321 WARN DFSUtil - Unexpected value for data transfer bytes=42 duration=0
17:38:00.371 INFO BlockManagerInfo - Removed broadcast_111_piece0 on localhost:37953 in memory (size: 228.0 B, free: 1919.6 MiB)
17:38:00.371 INFO Executor - Finished task 0.0 in stage 66.0 (TID 104). 154144 bytes result sent to driver
17:38:00.372 INFO BlockManagerInfo - Removed broadcast_113_piece0 on localhost:37953 in memory (size: 1473.0 B, free: 1919.6 MiB)
17:38:00.372 INFO TaskSetManager - Finished task 0.0 in stage 66.0 (TID 104) in 81 ms on localhost (executor driver) (1/1)
17:38:00.372 INFO TaskSchedulerImpl - Removed TaskSet 66.0, whose tasks have all completed, from pool
17:38:00.373 INFO DAGScheduler - ResultStage 66 (collect at ReadsSparkSinkUnitTest.java:182) finished in 0.094 s
17:38:00.373 INFO DAGScheduler - Job 49 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:00.373 INFO TaskSchedulerImpl - Killing all running tasks in stage 66: Stage finished
17:38:00.373 INFO BlockManagerInfo - Removed broadcast_115_piece0 on localhost:37953 in memory (size: 107.3 KiB, free: 1919.7 MiB)
17:38:00.373 INFO DAGScheduler - Job 49 finished: collect at ReadsSparkSinkUnitTest.java:182, took 0.095617 s
17:38:00.376 INFO BlockManagerInfo - Removed broadcast_112_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.7 MiB)
17:38:00.379 INFO BlockManagerInfo - Removed broadcast_114_piece0 on localhost:37953 in memory (size: 1473.0 B, free: 1919.7 MiB)
17:38:00.379 INFO SparkContext - Starting job: count at ReadsSparkSinkUnitTest.java:185
17:38:00.380 INFO DAGScheduler - Got job 50 (count at ReadsSparkSinkUnitTest.java:185) with 1 output partitions
17:38:00.380 INFO DAGScheduler - Final stage: ResultStage 67 (count at ReadsSparkSinkUnitTest.java:185)
17:38:00.380 INFO DAGScheduler - Parents of final stage: List()
17:38:00.380 INFO DAGScheduler - Missing parents: List()
17:38:00.380 INFO DAGScheduler - Submitting ResultStage 67 (MapPartitionsRDD[246] at filter at ReadsSparkSource.java:96), which has no missing parents
17:38:00.380 INFO BlockManagerInfo - Removed broadcast_116_piece0 on localhost:37953 in memory (size: 58.1 KiB, free: 1919.8 MiB)
17:38:00.391 INFO MemoryStore - Block broadcast_120 stored as values in memory (estimated size 286.8 KiB, free 1918.7 MiB)
17:38:00.392 INFO MemoryStore - Block broadcast_120_piece0 stored as bytes in memory (estimated size 103.6 KiB, free 1918.6 MiB)
17:38:00.392 INFO BlockManagerInfo - Added broadcast_120_piece0 in memory on localhost:37953 (size: 103.6 KiB, free: 1919.7 MiB)
17:38:00.392 INFO SparkContext - Created broadcast 120 from broadcast at DAGScheduler.scala:1580
17:38:00.392 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 67 (MapPartitionsRDD[246] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:38:00.392 INFO TaskSchedulerImpl - Adding task set 67.0 with 1 tasks resource profile 0
17:38:00.396 INFO TaskSetManager - Starting task 0.0 in stage 67.0 (TID 105) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7880 bytes)
17:38:00.397 INFO Executor - Running task 0.0 in stage 67.0 (TID 105)
17:38:00.420 INFO NewHadoopRDD - Input split: file:/home/runner/work/gatk/gatk/src/test/resources/org/broadinstitute/hellbender/tools/BQSR/NA12878.chr17_69k_70k.dictFix.cram:0+50619
17:38:00.429 INFO Executor - Finished task 0.0 in stage 67.0 (TID 105). 989 bytes result sent to driver
17:38:00.429 INFO TaskSetManager - Finished task 0.0 in stage 67.0 (TID 105) in 36 ms on localhost (executor driver) (1/1)
17:38:00.429 INFO DAGScheduler - ResultStage 67 (count at ReadsSparkSinkUnitTest.java:185) finished in 0.049 s
17:38:00.429 INFO DAGScheduler - Job 50 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:00.429 INFO TaskSchedulerImpl - Removed TaskSet 67.0, whose tasks have all completed, from pool
17:38:00.430 INFO TaskSchedulerImpl - Killing all running tasks in stage 67: Stage finished
17:38:00.430 INFO DAGScheduler - Job 50 finished: count at ReadsSparkSinkUnitTest.java:185, took 0.050129 s
17:38:00.433 INFO SparkContext - Starting job: count at ReadsSparkSinkUnitTest.java:185
17:38:00.433 INFO DAGScheduler - Got job 51 (count at ReadsSparkSinkUnitTest.java:185) with 1 output partitions
17:38:00.433 INFO DAGScheduler - Final stage: ResultStage 68 (count at ReadsSparkSinkUnitTest.java:185)
17:38:00.433 INFO DAGScheduler - Parents of final stage: List()
17:38:00.433 INFO DAGScheduler - Missing parents: List()
17:38:00.433 INFO DAGScheduler - Submitting ResultStage 68 (MapPartitionsRDD[263] at filter at ReadsSparkSource.java:96), which has no missing parents
17:38:00.444 INFO MemoryStore - Block broadcast_121 stored as values in memory (estimated size 286.8 KiB, free 1918.3 MiB)
17:38:00.445 INFO MemoryStore - Block broadcast_121_piece0 stored as bytes in memory (estimated size 103.6 KiB, free 1918.2 MiB)
17:38:00.445 INFO BlockManagerInfo - Added broadcast_121_piece0 in memory on localhost:37953 (size: 103.6 KiB, free: 1919.6 MiB)
17:38:00.445 INFO SparkContext - Created broadcast 121 from broadcast at DAGScheduler.scala:1580
17:38:00.446 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 68 (MapPartitionsRDD[263] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:38:00.446 INFO TaskSchedulerImpl - Adding task set 68.0 with 1 tasks resource profile 0
17:38:00.446 INFO TaskSetManager - Starting task 0.0 in stage 68.0 (TID 106) (localhost, executor driver, partition 0, ANY, 7853 bytes)
17:38:00.447 INFO Executor - Running task 0.0 in stage 68.0 (TID 106)
17:38:00.470 INFO NewHadoopRDD - Input split: hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest5_f0550c4d-35fe-4b14-841a-b6a8c2c2b6b9.cram:0+43713
17:38:00.471 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest5_f0550c4d-35fe-4b14-841a-b6a8c2c2b6b9.cram dst=null perm=null proto=rpc
17:38:00.472 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest5_f0550c4d-35fe-4b14-841a-b6a8c2c2b6b9.cram dst=null perm=null proto=rpc
17:38:00.473 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest5_f0550c4d-35fe-4b14-841a-b6a8c2c2b6b9.cram.crai dst=null perm=null proto=rpc
17:38:00.473 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest5_f0550c4d-35fe-4b14-841a-b6a8c2c2b6b9.crai dst=null perm=null proto=rpc
17:38:00.476 WARN DFSUtil - Unexpected value for data transfer bytes=1024 duration=0
17:38:00.476 WARN DFSUtil - Unexpected value for data transfer bytes=42995 duration=0
17:38:00.477 WARN DFSUtil - Unexpected value for data transfer bytes=42 duration=0
17:38:00.491 INFO Executor - Finished task 0.0 in stage 68.0 (TID 106). 989 bytes result sent to driver
17:38:00.492 INFO TaskSetManager - Finished task 0.0 in stage 68.0 (TID 106) in 46 ms on localhost (executor driver) (1/1)
17:38:00.492 INFO TaskSchedulerImpl - Removed TaskSet 68.0, whose tasks have all completed, from pool
17:38:00.492 INFO DAGScheduler - ResultStage 68 (count at ReadsSparkSinkUnitTest.java:185) finished in 0.058 s
17:38:00.492 INFO DAGScheduler - Job 51 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:00.492 INFO TaskSchedulerImpl - Killing all running tasks in stage 68: Stage finished
17:38:00.493 INFO DAGScheduler - Job 51 finished: count at ReadsSparkSinkUnitTest.java:185, took 0.059881 s
17:38:00.494 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:00.497 INFO MemoryStore - Block broadcast_122 stored as values in memory (estimated size 297.9 KiB, free 1917.9 MiB)
17:38:00.502 INFO MemoryStore - Block broadcast_122_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1917.8 MiB)
17:38:00.502 INFO BlockManagerInfo - Added broadcast_122_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.5 MiB)
17:38:00.503 INFO SparkContext - Created broadcast 122 from newAPIHadoopFile at PathSplitSource.java:96
17:38:00.522 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:00.525 INFO MemoryStore - Block broadcast_123 stored as values in memory (estimated size 297.9 KiB, free 1917.5 MiB)
17:38:00.530 INFO MemoryStore - Block broadcast_123_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1917.5 MiB)
17:38:00.531 INFO BlockManagerInfo - Added broadcast_123_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.5 MiB)
17:38:00.531 INFO SparkContext - Created broadcast 123 from newAPIHadoopFile at PathSplitSource.java:96
17:38:00.550 INFO FileInputFormat - Total input files to process : 1
17:38:00.552 INFO MemoryStore - Block broadcast_124 stored as values in memory (estimated size 160.7 KiB, free 1917.3 MiB)
17:38:00.553 INFO MemoryStore - Block broadcast_124_piece0 stored as bytes in memory (estimated size 9.6 KiB, free 1917.3 MiB)
17:38:00.553 INFO BlockManagerInfo - Added broadcast_124_piece0 in memory on localhost:37953 (size: 9.6 KiB, free: 1919.5 MiB)
17:38:00.553 INFO SparkContext - Created broadcast 124 from broadcast at ReadsSparkSink.java:133
17:38:00.567 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest6_ad233f06-2677-4777-84d7-fd3b0db7e696.sam.parts dst=null perm=null proto=rpc
17:38:00.568 INFO deprecation - mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
17:38:00.569 INFO HadoopMapRedCommitProtocol - Using output committer class org.apache.hadoop.mapred.FileOutputCommitter
17:38:00.569 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:00.569 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:00.570 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=mkdirs src=/user/runner/ReadsSparkSinkUnitTest6_ad233f06-2677-4777-84d7-fd3b0db7e696.sam.parts/_temporary/0 dst=null perm=runner:supergroup:rwxr-xr-x proto=rpc
17:38:00.578 INFO SparkContext - Starting job: runJob at SparkHadoopWriter.scala:83
17:38:00.578 INFO DAGScheduler - Registering RDD 277 (mapToPair at SparkUtils.java:161) as input to shuffle 15
17:38:00.578 INFO DAGScheduler - Got job 52 (runJob at SparkHadoopWriter.scala:83) with 1 output partitions
17:38:00.578 INFO DAGScheduler - Final stage: ResultStage 70 (runJob at SparkHadoopWriter.scala:83)
17:38:00.578 INFO DAGScheduler - Parents of final stage: List(ShuffleMapStage 69)
17:38:00.579 INFO DAGScheduler - Missing parents: List(ShuffleMapStage 69)
17:38:00.579 INFO DAGScheduler - Submitting ShuffleMapStage 69 (MapPartitionsRDD[277] at mapToPair at SparkUtils.java:161), which has no missing parents
17:38:00.595 INFO MemoryStore - Block broadcast_125 stored as values in memory (estimated size 520.4 KiB, free 1916.8 MiB)
17:38:00.596 INFO MemoryStore - Block broadcast_125_piece0 stored as bytes in memory (estimated size 166.1 KiB, free 1916.7 MiB)
17:38:00.596 INFO BlockManagerInfo - Added broadcast_125_piece0 in memory on localhost:37953 (size: 166.1 KiB, free: 1919.3 MiB)
17:38:00.596 INFO SparkContext - Created broadcast 125 from broadcast at DAGScheduler.scala:1580
17:38:00.597 INFO DAGScheduler - Submitting 1 missing tasks from ShuffleMapStage 69 (MapPartitionsRDD[277] at mapToPair at SparkUtils.java:161) (first 15 tasks are for partitions Vector(0))
17:38:00.597 INFO TaskSchedulerImpl - Adding task set 69.0 with 1 tasks resource profile 0
17:38:00.597 INFO TaskSetManager - Starting task 0.0 in stage 69.0 (TID 107) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7861 bytes)
17:38:00.598 INFO Executor - Running task 0.0 in stage 69.0 (TID 107)
17:38:00.599 INFO FsDatasetAsyncDiskService - Scheduling blk_1073741860_1036 replica FinalizedReplica, blk_1073741860_1036, FINALIZED
getNumBytes() = 204
getBytesOnDisk() = 204
getVisibleLength()= 204
getVolume() = /tmp/minicluster_storage14473397491642437130/data/data2
getBlockURI() = file:/tmp/minicluster_storage14473397491642437130/data/data2/current/BP-617366194-10.1.0.252-1761586664431/current/finalized/subdir0/subdir0/blk_1073741860 for deletion
17:38:00.599 INFO FsDatasetAsyncDiskService - Scheduling blk_1073741861_1037 replica FinalizedReplica, blk_1073741861_1037, FINALIZED
getNumBytes() = 592
getBytesOnDisk() = 592
getVisibleLength()= 592
getVolume() = /tmp/minicluster_storage14473397491642437130/data/data1
getBlockURI() = file:/tmp/minicluster_storage14473397491642437130/data/data1/current/BP-617366194-10.1.0.252-1761586664431/current/finalized/subdir0/subdir0/blk_1073741861 for deletion
17:38:00.600 INFO FsDatasetAsyncDiskService - Scheduling blk_1073741854_1030 replica FinalizedReplica, blk_1073741854_1030, FINALIZED
getNumBytes() = 5472
getBytesOnDisk() = 5472
getVisibleLength()= 5472
getVolume() = /tmp/minicluster_storage14473397491642437130/data/data2
getBlockURI() = file:/tmp/minicluster_storage14473397491642437130/data/data2/current/BP-617366194-10.1.0.252-1761586664431/current/finalized/subdir0/subdir0/blk_1073741854 for deletion
17:38:00.600 INFO FsDatasetAsyncDiskService - Deleted BP-617366194-10.1.0.252-1761586664431 blk_1073741860_1036 URI file:/tmp/minicluster_storage14473397491642437130/data/data2/current/BP-617366194-10.1.0.252-1761586664431/current/finalized/subdir0/subdir0/blk_1073741860
17:38:00.600 INFO FsDatasetAsyncDiskService - Deleted BP-617366194-10.1.0.252-1761586664431 blk_1073741861_1037 URI file:/tmp/minicluster_storage14473397491642437130/data/data1/current/BP-617366194-10.1.0.252-1761586664431/current/finalized/subdir0/subdir0/blk_1073741861
17:38:00.600 INFO FsDatasetAsyncDiskService - Deleted BP-617366194-10.1.0.252-1761586664431 blk_1073741854_1030 URI file:/tmp/minicluster_storage14473397491642437130/data/data2/current/BP-617366194-10.1.0.252-1761586664431/current/finalized/subdir0/subdir0/blk_1073741854
17:38:00.632 INFO NewHadoopRDD - Input split: file:/home/runner/work/gatk/gatk/src/test/resources/org/broadinstitute/hellbender/tools/BQSR/HiSeq.1mb.1RG.2k_lines.bam:0+222075
17:38:00.633 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:00.635 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:00.643 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:00.653 INFO Executor - Finished task 0.0 in stage 69.0 (TID 107). 1148 bytes result sent to driver
17:38:00.654 INFO TaskSetManager - Finished task 0.0 in stage 69.0 (TID 107) in 57 ms on localhost (executor driver) (1/1)
17:38:00.654 INFO TaskSchedulerImpl - Removed TaskSet 69.0, whose tasks have all completed, from pool
17:38:00.654 INFO DAGScheduler - ShuffleMapStage 69 (mapToPair at SparkUtils.java:161) finished in 0.075 s
17:38:00.654 INFO DAGScheduler - looking for newly runnable stages
17:38:00.654 INFO DAGScheduler - running: HashSet()
17:38:00.654 INFO DAGScheduler - waiting: HashSet(ResultStage 70)
17:38:00.654 INFO DAGScheduler - failed: HashSet()
17:38:00.655 INFO DAGScheduler - Submitting ResultStage 70 (MapPartitionsRDD[283] at saveAsTextFile at SamSink.java:65), which has no missing parents
17:38:00.661 INFO MemoryStore - Block broadcast_126 stored as values in memory (estimated size 241.1 KiB, free 1916.4 MiB)
17:38:00.662 INFO MemoryStore - Block broadcast_126_piece0 stored as bytes in memory (estimated size 67.0 KiB, free 1916.4 MiB)
17:38:00.662 INFO BlockManagerInfo - Added broadcast_126_piece0 in memory on localhost:37953 (size: 67.0 KiB, free: 1919.3 MiB)
17:38:00.662 INFO SparkContext - Created broadcast 126 from broadcast at DAGScheduler.scala:1580
17:38:00.663 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 70 (MapPartitionsRDD[283] at saveAsTextFile at SamSink.java:65) (first 15 tasks are for partitions Vector(0))
17:38:00.663 INFO TaskSchedulerImpl - Adding task set 70.0 with 1 tasks resource profile 0
17:38:00.663 INFO TaskSetManager - Starting task 0.0 in stage 70.0 (TID 108) (localhost, executor driver, partition 0, NODE_LOCAL, 7513 bytes)
17:38:00.664 INFO Executor - Running task 0.0 in stage 70.0 (TID 108)
17:38:00.670 INFO ShuffleBlockFetcherIterator - Getting 1 (343.8 KiB) non-empty blocks including 1 (343.8 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
17:38:00.670 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
17:38:00.684 INFO HadoopMapRedCommitProtocol - Using output committer class org.apache.hadoop.mapred.FileOutputCommitter
17:38:00.684 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:00.685 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:00.686 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest6_ad233f06-2677-4777-84d7-fd3b0db7e696.sam.parts/_temporary/0/_temporary/attempt_202510271738008963169821661170358_0283_m_000000_0/part-00000 dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:00.689 INFO StateChange - BLOCK* allocate blk_1073741869_1045, replicas=127.0.0.1:37927 for /user/runner/ReadsSparkSinkUnitTest6_ad233f06-2677-4777-84d7-fd3b0db7e696.sam.parts/_temporary/0/_temporary/attempt_202510271738008963169821661170358_0283_m_000000_0/part-00000
17:38:00.693 INFO DataNode - Receiving BP-617366194-10.1.0.252-1761586664431:blk_1073741869_1045 src: /127.0.0.1:54626 dest: /127.0.0.1:37927
17:38:00.696 INFO clienttrace - src: /127.0.0.1:54626, dest: /127.0.0.1:37927, bytes: 761729, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1369393814_1, offset: 0, srvID: 379dd5e5-59c3-42cd-b9d3-67ca5e44560b, blockid: BP-617366194-10.1.0.252-1761586664431:blk_1073741869_1045, duration(ns): 2110855
17:38:00.696 INFO DataNode - PacketResponder: BP-617366194-10.1.0.252-1761586664431:blk_1073741869_1045, type=LAST_IN_PIPELINE terminating
17:38:00.697 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest6_ad233f06-2677-4777-84d7-fd3b0db7e696.sam.parts/_temporary/0/_temporary/attempt_202510271738008963169821661170358_0283_m_000000_0/part-00000 is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:38:00.697 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest6_ad233f06-2677-4777-84d7-fd3b0db7e696.sam.parts/_temporary/0/_temporary/attempt_202510271738008963169821661170358_0283_m_000000_0 dst=null perm=null proto=rpc
17:38:00.698 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest6_ad233f06-2677-4777-84d7-fd3b0db7e696.sam.parts/_temporary/0/_temporary/attempt_202510271738008963169821661170358_0283_m_000000_0 dst=null perm=null proto=rpc
17:38:00.698 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest6_ad233f06-2677-4777-84d7-fd3b0db7e696.sam.parts/_temporary/0/task_202510271738008963169821661170358_0283_m_000000 dst=null perm=null proto=rpc
17:38:00.699 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=rename src=/user/runner/ReadsSparkSinkUnitTest6_ad233f06-2677-4777-84d7-fd3b0db7e696.sam.parts/_temporary/0/_temporary/attempt_202510271738008963169821661170358_0283_m_000000_0 dst=/user/runner/ReadsSparkSinkUnitTest6_ad233f06-2677-4777-84d7-fd3b0db7e696.sam.parts/_temporary/0/task_202510271738008963169821661170358_0283_m_000000 perm=runner:supergroup:rwxr-xr-x proto=rpc
17:38:00.699 INFO FileOutputCommitter - Saved output of task 'attempt_202510271738008963169821661170358_0283_m_000000_0' to hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest6_ad233f06-2677-4777-84d7-fd3b0db7e696.sam.parts/_temporary/0/task_202510271738008963169821661170358_0283_m_000000
17:38:00.699 INFO SparkHadoopMapRedUtil - attempt_202510271738008963169821661170358_0283_m_000000_0: Committed. Elapsed time: 1 ms.
17:38:00.700 INFO Executor - Finished task 0.0 in stage 70.0 (TID 108). 1858 bytes result sent to driver
17:38:00.700 INFO TaskSetManager - Finished task 0.0 in stage 70.0 (TID 108) in 37 ms on localhost (executor driver) (1/1)
17:38:00.700 INFO TaskSchedulerImpl - Removed TaskSet 70.0, whose tasks have all completed, from pool
17:38:00.701 INFO DAGScheduler - ResultStage 70 (runJob at SparkHadoopWriter.scala:83) finished in 0.046 s
17:38:00.701 INFO DAGScheduler - Job 52 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:00.701 INFO TaskSchedulerImpl - Killing all running tasks in stage 70: Stage finished
17:38:00.701 INFO DAGScheduler - Job 52 finished: runJob at SparkHadoopWriter.scala:83, took 0.123211 s
17:38:00.701 INFO SparkHadoopWriter - Start to commit write Job job_202510271738008963169821661170358_0283.
17:38:00.702 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=listStatus src=/user/runner/ReadsSparkSinkUnitTest6_ad233f06-2677-4777-84d7-fd3b0db7e696.sam.parts/_temporary/0 dst=null perm=null proto=rpc
17:38:00.703 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest6_ad233f06-2677-4777-84d7-fd3b0db7e696.sam.parts dst=null perm=null proto=rpc
17:38:00.703 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=listStatus src=/user/runner/ReadsSparkSinkUnitTest6_ad233f06-2677-4777-84d7-fd3b0db7e696.sam.parts/_temporary/0/task_202510271738008963169821661170358_0283_m_000000 dst=null perm=null proto=rpc
17:38:00.704 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest6_ad233f06-2677-4777-84d7-fd3b0db7e696.sam.parts/part-00000 dst=null perm=null proto=rpc
17:38:00.704 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=rename src=/user/runner/ReadsSparkSinkUnitTest6_ad233f06-2677-4777-84d7-fd3b0db7e696.sam.parts/_temporary/0/task_202510271738008963169821661170358_0283_m_000000/part-00000 dst=/user/runner/ReadsSparkSinkUnitTest6_ad233f06-2677-4777-84d7-fd3b0db7e696.sam.parts/part-00000 perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:00.705 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=delete src=/user/runner/ReadsSparkSinkUnitTest6_ad233f06-2677-4777-84d7-fd3b0db7e696.sam.parts/_temporary dst=null perm=null proto=rpc
17:38:00.706 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest6_ad233f06-2677-4777-84d7-fd3b0db7e696.sam.parts/_SUCCESS dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:00.706 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest6_ad233f06-2677-4777-84d7-fd3b0db7e696.sam.parts/_SUCCESS is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:38:00.707 INFO audit - allowed=false ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=delete src=/user/runner/ReadsSparkSinkUnitTest6_ad233f06-2677-4777-84d7-fd3b0db7e696.sam.parts/.spark-staging-283 dst=null perm=null proto=rpc
17:38:00.707 INFO SparkHadoopWriter - Write Job job_202510271738008963169821661170358_0283 committed. Elapsed time: 5 ms.
17:38:00.708 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest6_ad233f06-2677-4777-84d7-fd3b0db7e696.sam.parts/header dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:00.710 INFO StateChange - BLOCK* allocate blk_1073741870_1046, replicas=127.0.0.1:37927 for /user/runner/ReadsSparkSinkUnitTest6_ad233f06-2677-4777-84d7-fd3b0db7e696.sam.parts/header
17:38:00.711 INFO DataNode - Receiving BP-617366194-10.1.0.252-1761586664431:blk_1073741870_1046 src: /127.0.0.1:54636 dest: /127.0.0.1:37927
17:38:00.713 INFO clienttrace - src: /127.0.0.1:54636, dest: /127.0.0.1:37927, bytes: 85829, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1369393814_1, offset: 0, srvID: 379dd5e5-59c3-42cd-b9d3-67ca5e44560b, blockid: BP-617366194-10.1.0.252-1761586664431:blk_1073741870_1046, duration(ns): 680173
17:38:00.713 INFO DataNode - PacketResponder: BP-617366194-10.1.0.252-1761586664431:blk_1073741870_1046, type=LAST_IN_PIPELINE terminating
17:38:00.716 INFO FSNamesystem - BLOCK* blk_1073741870_1046 is COMMITTED but not COMPLETE(numNodes= 0 < minimum = 1) in file /user/runner/ReadsSparkSinkUnitTest6_ad233f06-2677-4777-84d7-fd3b0db7e696.sam.parts/header
17:38:01.117 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest6_ad233f06-2677-4777-84d7-fd3b0db7e696.sam.parts/header is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:38:01.118 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=listStatus src=/user/runner/ReadsSparkSinkUnitTest6_ad233f06-2677-4777-84d7-fd3b0db7e696.sam.parts dst=null perm=null proto=rpc
17:38:01.119 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest6_ad233f06-2677-4777-84d7-fd3b0db7e696.sam.parts/output dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:01.120 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest6_ad233f06-2677-4777-84d7-fd3b0db7e696.sam.parts/output is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:38:01.120 INFO HadoopFileSystemWrapper - Concatenating 2 parts to hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest6_ad233f06-2677-4777-84d7-fd3b0db7e696.sam
17:38:01.121 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=concat src=[/user/runner/ReadsSparkSinkUnitTest6_ad233f06-2677-4777-84d7-fd3b0db7e696.sam.parts/header, /user/runner/ReadsSparkSinkUnitTest6_ad233f06-2677-4777-84d7-fd3b0db7e696.sam.parts/part-00000] dst=/user/runner/ReadsSparkSinkUnitTest6_ad233f06-2677-4777-84d7-fd3b0db7e696.sam.parts/output perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:01.121 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest6_ad233f06-2677-4777-84d7-fd3b0db7e696.sam dst=null perm=null proto=rpc
17:38:01.122 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=rename src=/user/runner/ReadsSparkSinkUnitTest6_ad233f06-2677-4777-84d7-fd3b0db7e696.sam.parts/output dst=/user/runner/ReadsSparkSinkUnitTest6_ad233f06-2677-4777-84d7-fd3b0db7e696.sam perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:01.122 INFO HadoopFileSystemWrapper - Concatenating to hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest6_ad233f06-2677-4777-84d7-fd3b0db7e696.sam done
17:38:01.123 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=delete src=/user/runner/ReadsSparkSinkUnitTest6_ad233f06-2677-4777-84d7-fd3b0db7e696.sam.parts dst=null perm=null proto=rpc
17:38:01.123 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest6_ad233f06-2677-4777-84d7-fd3b0db7e696.sam dst=null perm=null proto=rpc
17:38:01.124 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest6_ad233f06-2677-4777-84d7-fd3b0db7e696.sam dst=null perm=null proto=rpc
17:38:01.125 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest6_ad233f06-2677-4777-84d7-fd3b0db7e696.sam dst=null perm=null proto=rpc
17:38:01.125 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest6_ad233f06-2677-4777-84d7-fd3b0db7e696.sam dst=null perm=null proto=rpc
WARNING 2025-10-27 17:38:01 SamReaderFactory Unable to detect file format from input URL or stream, assuming SAM format.
17:38:01.128 WARN DFSUtil - Unexpected value for data transfer bytes=86501 duration=0
17:38:01.130 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest6_ad233f06-2677-4777-84d7-fd3b0db7e696.sam dst=null perm=null proto=rpc
17:38:01.130 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest6_ad233f06-2677-4777-84d7-fd3b0db7e696.sam dst=null perm=null proto=rpc
17:38:01.131 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest6_ad233f06-2677-4777-84d7-fd3b0db7e696.sam dst=null perm=null proto=rpc
WARNING 2025-10-27 17:38:01 SamReaderFactory Unable to detect file format from input URL or stream, assuming SAM format.
17:38:01.134 WARN DFSUtil - Unexpected value for data transfer bytes=767681 duration=0
17:38:01.135 INFO MemoryStore - Block broadcast_127 stored as values in memory (estimated size 160.7 KiB, free 1916.2 MiB)
17:38:01.136 INFO MemoryStore - Block broadcast_127_piece0 stored as bytes in memory (estimated size 9.6 KiB, free 1916.2 MiB)
17:38:01.136 INFO BlockManagerInfo - Added broadcast_127_piece0 in memory on localhost:37953 (size: 9.6 KiB, free: 1919.3 MiB)
17:38:01.136 INFO SparkContext - Created broadcast 127 from broadcast at SamSource.java:78
17:38:01.138 INFO MemoryStore - Block broadcast_128 stored as values in memory (estimated size 297.9 KiB, free 1915.9 MiB)
17:38:01.145 INFO BlockManagerInfo - Removed broadcast_123_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.3 MiB)
17:38:01.146 INFO BlockManagerInfo - Removed broadcast_109_piece0 on localhost:37953 in memory (size: 228.0 B, free: 1919.3 MiB)
17:38:01.147 INFO BlockManagerInfo - Removed broadcast_119_piece0 on localhost:37953 in memory (size: 103.6 KiB, free: 1919.4 MiB)
17:38:01.147 INFO BlockManagerInfo - Removed broadcast_124_piece0 on localhost:37953 in memory (size: 9.6 KiB, free: 1919.4 MiB)
17:38:01.148 INFO BlockManagerInfo - Removed broadcast_121_piece0 on localhost:37953 in memory (size: 103.6 KiB, free: 1919.5 MiB)
17:38:01.148 INFO MemoryStore - Block broadcast_128_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1917.1 MiB)
17:38:01.148 INFO BlockManagerInfo - Removed broadcast_118_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.6 MiB)
17:38:01.149 INFO BlockManagerInfo - Added broadcast_128_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.5 MiB)
17:38:01.149 INFO SparkContext - Created broadcast 128 from newAPIHadoopFile at SamSource.java:108
17:38:01.149 INFO BlockManagerInfo - Removed broadcast_126_piece0 on localhost:37953 in memory (size: 67.0 KiB, free: 1919.6 MiB)
17:38:01.151 INFO BlockManagerInfo - Removed broadcast_117_piece0 on localhost:37953 in memory (size: 187.0 B, free: 1919.6 MiB)
17:38:01.152 INFO BlockManagerInfo - Removed broadcast_120_piece0 on localhost:37953 in memory (size: 103.6 KiB, free: 1919.7 MiB)
17:38:01.153 INFO BlockManagerInfo - Removed broadcast_110_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.7 MiB)
17:38:01.153 INFO BlockManagerInfo - Removed broadcast_125_piece0 on localhost:37953 in memory (size: 166.1 KiB, free: 1919.9 MiB)
17:38:01.156 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest6_ad233f06-2677-4777-84d7-fd3b0db7e696.sam dst=null perm=null proto=rpc
17:38:01.156 INFO FileInputFormat - Total input files to process : 1
17:38:01.157 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest6_ad233f06-2677-4777-84d7-fd3b0db7e696.sam dst=null perm=null proto=rpc
17:38:01.168 INFO SparkContext - Starting job: collect at ReadsSparkSinkUnitTest.java:182
17:38:01.168 INFO DAGScheduler - Got job 53 (collect at ReadsSparkSinkUnitTest.java:182) with 1 output partitions
17:38:01.168 INFO DAGScheduler - Final stage: ResultStage 71 (collect at ReadsSparkSinkUnitTest.java:182)
17:38:01.168 INFO DAGScheduler - Parents of final stage: List()
17:38:01.168 INFO DAGScheduler - Missing parents: List()
17:38:01.169 INFO DAGScheduler - Submitting ResultStage 71 (MapPartitionsRDD[288] at filter at ReadsSparkSource.java:96), which has no missing parents
17:38:01.169 INFO MemoryStore - Block broadcast_129 stored as values in memory (estimated size 7.5 KiB, free 1919.1 MiB)
17:38:01.170 INFO MemoryStore - Block broadcast_129_piece0 stored as bytes in memory (estimated size 3.8 KiB, free 1919.1 MiB)
17:38:01.170 INFO BlockManagerInfo - Added broadcast_129_piece0 in memory on localhost:37953 (size: 3.8 KiB, free: 1919.9 MiB)
17:38:01.170 INFO SparkContext - Created broadcast 129 from broadcast at DAGScheduler.scala:1580
17:38:01.170 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 71 (MapPartitionsRDD[288] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:38:01.170 INFO TaskSchedulerImpl - Adding task set 71.0 with 1 tasks resource profile 0
17:38:01.171 INFO TaskSetManager - Starting task 0.0 in stage 71.0 (TID 109) (localhost, executor driver, partition 0, ANY, 7852 bytes)
17:38:01.172 INFO Executor - Running task 0.0 in stage 71.0 (TID 109)
17:38:01.173 INFO NewHadoopRDD - Input split: hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest6_ad233f06-2677-4777-84d7-fd3b0db7e696.sam:0+847558
17:38:01.178 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest6_ad233f06-2677-4777-84d7-fd3b0db7e696.sam dst=null perm=null proto=rpc
17:38:01.184 WARN DFSUtil - Unexpected value for data transfer bytes=86501 duration=0
17:38:01.206 WARN DFSUtil - Unexpected value for data transfer bytes=767681 duration=0
17:38:01.213 INFO Executor - Finished task 0.0 in stage 71.0 (TID 109). 651526 bytes result sent to driver
17:38:01.215 INFO TaskSetManager - Finished task 0.0 in stage 71.0 (TID 109) in 44 ms on localhost (executor driver) (1/1)
17:38:01.215 INFO TaskSchedulerImpl - Removed TaskSet 71.0, whose tasks have all completed, from pool
17:38:01.215 INFO DAGScheduler - ResultStage 71 (collect at ReadsSparkSinkUnitTest.java:182) finished in 0.046 s
17:38:01.215 INFO DAGScheduler - Job 53 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:01.215 INFO TaskSchedulerImpl - Killing all running tasks in stage 71: Stage finished
17:38:01.215 INFO DAGScheduler - Job 53 finished: collect at ReadsSparkSinkUnitTest.java:182, took 0.047159 s
17:38:01.225 INFO SparkContext - Starting job: count at ReadsSparkSinkUnitTest.java:185
17:38:01.226 INFO DAGScheduler - Got job 54 (count at ReadsSparkSinkUnitTest.java:185) with 1 output partitions
17:38:01.226 INFO DAGScheduler - Final stage: ResultStage 72 (count at ReadsSparkSinkUnitTest.java:185)
17:38:01.226 INFO DAGScheduler - Parents of final stage: List()
17:38:01.226 INFO DAGScheduler - Missing parents: List()
17:38:01.226 INFO DAGScheduler - Submitting ResultStage 72 (MapPartitionsRDD[270] at filter at ReadsSparkSource.java:96), which has no missing parents
17:38:01.242 INFO MemoryStore - Block broadcast_130 stored as values in memory (estimated size 426.1 KiB, free 1918.7 MiB)
17:38:01.243 INFO MemoryStore - Block broadcast_130_piece0 stored as bytes in memory (estimated size 153.6 KiB, free 1918.6 MiB)
17:38:01.243 INFO BlockManagerInfo - Added broadcast_130_piece0 in memory on localhost:37953 (size: 153.6 KiB, free: 1919.7 MiB)
17:38:01.243 INFO SparkContext - Created broadcast 130 from broadcast at DAGScheduler.scala:1580
17:38:01.244 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 72 (MapPartitionsRDD[270] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:38:01.244 INFO TaskSchedulerImpl - Adding task set 72.0 with 1 tasks resource profile 0
17:38:01.244 INFO TaskSetManager - Starting task 0.0 in stage 72.0 (TID 110) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7872 bytes)
17:38:01.245 INFO Executor - Running task 0.0 in stage 72.0 (TID 110)
17:38:01.277 INFO NewHadoopRDD - Input split: file:/home/runner/work/gatk/gatk/src/test/resources/org/broadinstitute/hellbender/tools/BQSR/HiSeq.1mb.1RG.2k_lines.bam:0+222075
17:38:01.278 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:01.280 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:01.287 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:01.291 INFO Executor - Finished task 0.0 in stage 72.0 (TID 110). 989 bytes result sent to driver
17:38:01.291 INFO TaskSetManager - Finished task 0.0 in stage 72.0 (TID 110) in 47 ms on localhost (executor driver) (1/1)
17:38:01.291 INFO TaskSchedulerImpl - Removed TaskSet 72.0, whose tasks have all completed, from pool
17:38:01.291 INFO DAGScheduler - ResultStage 72 (count at ReadsSparkSinkUnitTest.java:185) finished in 0.065 s
17:38:01.292 INFO DAGScheduler - Job 54 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:01.292 INFO TaskSchedulerImpl - Killing all running tasks in stage 72: Stage finished
17:38:01.293 INFO DAGScheduler - Job 54 finished: count at ReadsSparkSinkUnitTest.java:185, took 0.067236 s
17:38:01.296 INFO SparkContext - Starting job: count at ReadsSparkSinkUnitTest.java:185
17:38:01.296 INFO DAGScheduler - Got job 55 (count at ReadsSparkSinkUnitTest.java:185) with 1 output partitions
17:38:01.296 INFO DAGScheduler - Final stage: ResultStage 73 (count at ReadsSparkSinkUnitTest.java:185)
17:38:01.296 INFO DAGScheduler - Parents of final stage: List()
17:38:01.296 INFO DAGScheduler - Missing parents: List()
17:38:01.296 INFO DAGScheduler - Submitting ResultStage 73 (MapPartitionsRDD[288] at filter at ReadsSparkSource.java:96), which has no missing parents
17:38:01.297 INFO MemoryStore - Block broadcast_131 stored as values in memory (estimated size 7.4 KiB, free 1918.6 MiB)
17:38:01.297 INFO MemoryStore - Block broadcast_131_piece0 stored as bytes in memory (estimated size 3.8 KiB, free 1918.6 MiB)
17:38:01.298 INFO BlockManagerInfo - Added broadcast_131_piece0 in memory on localhost:37953 (size: 3.8 KiB, free: 1919.7 MiB)
17:38:01.298 INFO SparkContext - Created broadcast 131 from broadcast at DAGScheduler.scala:1580
17:38:01.298 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 73 (MapPartitionsRDD[288] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:38:01.298 INFO TaskSchedulerImpl - Adding task set 73.0 with 1 tasks resource profile 0
17:38:01.299 INFO TaskSetManager - Starting task 0.0 in stage 73.0 (TID 111) (localhost, executor driver, partition 0, ANY, 7852 bytes)
17:38:01.299 INFO Executor - Running task 0.0 in stage 73.0 (TID 111)
17:38:01.302 INFO NewHadoopRDD - Input split: hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest6_ad233f06-2677-4777-84d7-fd3b0db7e696.sam:0+847558
17:38:01.303 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest6_ad233f06-2677-4777-84d7-fd3b0db7e696.sam dst=null perm=null proto=rpc
17:38:01.305 WARN DFSUtil - Unexpected value for data transfer bytes=86501 duration=0
17:38:01.315 INFO Executor - Finished task 0.0 in stage 73.0 (TID 111). 989 bytes result sent to driver
17:38:01.315 INFO TaskSetManager - Finished task 0.0 in stage 73.0 (TID 111) in 16 ms on localhost (executor driver) (1/1)
17:38:01.316 INFO TaskSchedulerImpl - Removed TaskSet 73.0, whose tasks have all completed, from pool
17:38:01.316 INFO DAGScheduler - ResultStage 73 (count at ReadsSparkSinkUnitTest.java:185) finished in 0.019 s
17:38:01.316 INFO DAGScheduler - Job 55 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:01.316 INFO TaskSchedulerImpl - Killing all running tasks in stage 73: Stage finished
17:38:01.316 INFO DAGScheduler - Job 55 finished: count at ReadsSparkSinkUnitTest.java:185, took 0.020703 s
17:38:01.318 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:01.321 INFO MemoryStore - Block broadcast_132 stored as values in memory (estimated size 297.9 KiB, free 1918.3 MiB)
17:38:01.326 INFO MemoryStore - Block broadcast_132_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1918.2 MiB)
17:38:01.326 INFO BlockManagerInfo - Added broadcast_132_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.7 MiB)
17:38:01.327 INFO SparkContext - Created broadcast 132 from newAPIHadoopFile at PathSplitSource.java:96
17:38:01.350 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:01.352 INFO MemoryStore - Block broadcast_133 stored as values in memory (estimated size 297.9 KiB, free 1917.9 MiB)
17:38:01.358 INFO MemoryStore - Block broadcast_133_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1917.9 MiB)
17:38:01.358 INFO BlockManagerInfo - Added broadcast_133_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.6 MiB)
17:38:01.359 INFO SparkContext - Created broadcast 133 from newAPIHadoopFile at PathSplitSource.java:96
17:38:01.377 INFO MemoryStore - Block broadcast_134 stored as values in memory (estimated size 160.7 KiB, free 1917.7 MiB)
17:38:01.378 INFO MemoryStore - Block broadcast_134_piece0 stored as bytes in memory (estimated size 9.6 KiB, free 1917.7 MiB)
17:38:01.379 INFO BlockManagerInfo - Added broadcast_134_piece0 in memory on localhost:37953 (size: 9.6 KiB, free: 1919.6 MiB)
17:38:01.379 INFO SparkContext - Created broadcast 134 from broadcast at ReadsSparkSink.java:133
17:38:01.381 INFO MemoryStore - Block broadcast_135 stored as values in memory (estimated size 163.2 KiB, free 1917.6 MiB)
17:38:01.382 INFO MemoryStore - Block broadcast_135_piece0 stored as bytes in memory (estimated size 9.6 KiB, free 1917.6 MiB)
17:38:01.382 INFO BlockManagerInfo - Added broadcast_135_piece0 in memory on localhost:37953 (size: 9.6 KiB, free: 1919.6 MiB)
17:38:01.382 INFO SparkContext - Created broadcast 135 from broadcast at AnySamSinkMultiple.java:80
17:38:01.387 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:01.387 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:01.387 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:01.397 INFO FileInputFormat - Total input files to process : 1
17:38:01.406 INFO SparkContext - Starting job: runJob at SparkHadoopWriter.scala:83
17:38:01.406 INFO DAGScheduler - Registering RDD 296 (repartition at ReadsSparkSinkUnitTest.java:210) as input to shuffle 16
17:38:01.406 INFO DAGScheduler - Got job 56 (runJob at SparkHadoopWriter.scala:83) with 2 output partitions
17:38:01.407 INFO DAGScheduler - Final stage: ResultStage 75 (runJob at SparkHadoopWriter.scala:83)
17:38:01.407 INFO DAGScheduler - Parents of final stage: List(ShuffleMapStage 74)
17:38:01.407 INFO DAGScheduler - Missing parents: List(ShuffleMapStage 74)
17:38:01.407 INFO DAGScheduler - Submitting ShuffleMapStage 74 (MapPartitionsRDD[296] at repartition at ReadsSparkSinkUnitTest.java:210), which has no missing parents
17:38:01.423 INFO MemoryStore - Block broadcast_136 stored as values in memory (estimated size 427.7 KiB, free 1917.1 MiB)
17:38:01.430 INFO BlockManagerInfo - Removed broadcast_130_piece0 on localhost:37953 in memory (size: 153.6 KiB, free: 1919.8 MiB)
17:38:01.430 INFO MemoryStore - Block broadcast_136_piece0 stored as bytes in memory (estimated size 154.6 KiB, free 1917.5 MiB)
17:38:01.430 INFO BlockManagerInfo - Added broadcast_136_piece0 in memory on localhost:37953 (size: 154.6 KiB, free: 1919.6 MiB)
17:38:01.430 INFO BlockManagerInfo - Removed broadcast_133_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.7 MiB)
17:38:01.431 INFO SparkContext - Created broadcast 136 from broadcast at DAGScheduler.scala:1580
17:38:01.431 INFO DAGScheduler - Submitting 1 missing tasks from ShuffleMapStage 74 (MapPartitionsRDD[296] at repartition at ReadsSparkSinkUnitTest.java:210) (first 15 tasks are for partitions Vector(0))
17:38:01.431 INFO TaskSchedulerImpl - Adding task set 74.0 with 1 tasks resource profile 0
17:38:01.431 INFO BlockManagerInfo - Removed broadcast_127_piece0 on localhost:37953 in memory (size: 9.6 KiB, free: 1919.7 MiB)
17:38:01.432 INFO BlockManagerInfo - Removed broadcast_128_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.7 MiB)
17:38:01.432 INFO TaskSetManager - Starting task 0.0 in stage 74.0 (TID 112) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7861 bytes)
17:38:01.432 INFO BlockManagerInfo - Removed broadcast_122_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.8 MiB)
17:38:01.433 INFO Executor - Running task 0.0 in stage 74.0 (TID 112)
17:38:01.436 INFO BlockManagerInfo - Removed broadcast_131_piece0 on localhost:37953 in memory (size: 3.8 KiB, free: 1919.8 MiB)
17:38:01.436 INFO BlockManagerInfo - Removed broadcast_129_piece0 on localhost:37953 in memory (size: 3.8 KiB, free: 1919.8 MiB)
17:38:01.466 INFO NewHadoopRDD - Input split: file:/home/runner/work/gatk/gatk/src/test/resources/org/broadinstitute/hellbender/tools/BQSR/HiSeq.1mb.1RG.2k_lines.bam:0+222075
17:38:01.468 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:01.469 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:01.476 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:01.489 INFO Executor - Finished task 0.0 in stage 74.0 (TID 112). 1149 bytes result sent to driver
17:38:01.489 INFO TaskSetManager - Finished task 0.0 in stage 74.0 (TID 112) in 57 ms on localhost (executor driver) (1/1)
17:38:01.489 INFO TaskSchedulerImpl - Removed TaskSet 74.0, whose tasks have all completed, from pool
17:38:01.489 INFO DAGScheduler - ShuffleMapStage 74 (repartition at ReadsSparkSinkUnitTest.java:210) finished in 0.082 s
17:38:01.489 INFO DAGScheduler - looking for newly runnable stages
17:38:01.489 INFO DAGScheduler - running: HashSet()
17:38:01.489 INFO DAGScheduler - waiting: HashSet(ResultStage 75)
17:38:01.489 INFO DAGScheduler - failed: HashSet()
17:38:01.490 INFO DAGScheduler - Submitting ResultStage 75 (MapPartitionsRDD[308] at mapToPair at AnySamSinkMultiple.java:89), which has no missing parents
17:38:01.497 INFO MemoryStore - Block broadcast_137 stored as values in memory (estimated size 150.2 KiB, free 1918.6 MiB)
17:38:01.497 INFO MemoryStore - Block broadcast_137_piece0 stored as bytes in memory (estimated size 56.2 KiB, free 1918.6 MiB)
17:38:01.497 INFO BlockManagerInfo - Added broadcast_137_piece0 in memory on localhost:37953 (size: 56.2 KiB, free: 1919.7 MiB)
17:38:01.498 INFO SparkContext - Created broadcast 137 from broadcast at DAGScheduler.scala:1580
17:38:01.498 INFO DAGScheduler - Submitting 2 missing tasks from ResultStage 75 (MapPartitionsRDD[308] at mapToPair at AnySamSinkMultiple.java:89) (first 15 tasks are for partitions Vector(0, 1))
17:38:01.498 INFO TaskSchedulerImpl - Adding task set 75.0 with 2 tasks resource profile 0
17:38:01.499 INFO TaskSetManager - Starting task 0.0 in stage 75.0 (TID 113) (localhost, executor driver, partition 0, NODE_LOCAL, 7860 bytes)
17:38:01.499 INFO TaskSetManager - Starting task 1.0 in stage 75.0 (TID 114) (localhost, executor driver, partition 1, NODE_LOCAL, 7860 bytes)
17:38:01.499 INFO Executor - Running task 1.0 in stage 75.0 (TID 114)
17:38:01.499 INFO Executor - Running task 0.0 in stage 75.0 (TID 113)
17:38:01.506 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:01.506 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:01.506 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:01.506 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:01.506 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:01.506 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:01.506 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:01.506 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:01.506 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:01.506 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:01.506 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:01.506 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:01.514 WARN IntelDeflaterFactory - Intel Deflater not supported, using Java.util.zip.Deflater
17:38:01.515 INFO ShuffleBlockFetcherIterator - Getting 1 (176.4 KiB) non-empty blocks including 1 (176.4 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
17:38:01.515 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
17:38:01.517 WARN IntelDeflaterFactory - Intel Deflater not supported, using Java.util.zip.Deflater
17:38:01.518 INFO ShuffleBlockFetcherIterator - Getting 1 (176.4 KiB) non-empty blocks including 1 (176.4 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
17:38:01.518 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
17:38:01.525 INFO FileOutputCommitter - Saved output of task 'attempt_202510271738013037080282302370339_0308_r_000000_0' to file:/tmp/ReadsSparkSinkUnitTest12034560671007073313.bam/_temporary/0/task_202510271738013037080282302370339_0308_r_000000
17:38:01.525 INFO SparkHadoopMapRedUtil - attempt_202510271738013037080282302370339_0308_r_000000_0: Committed. Elapsed time: 0 ms.
17:38:01.526 INFO Executor - Finished task 0.0 in stage 75.0 (TID 113). 1729 bytes result sent to driver
17:38:01.527 INFO TaskSetManager - Finished task 0.0 in stage 75.0 (TID 113) in 28 ms on localhost (executor driver) (1/2)
17:38:01.527 INFO FileOutputCommitter - Saved output of task 'attempt_202510271738013037080282302370339_0308_r_000001_0' to file:/tmp/ReadsSparkSinkUnitTest12034560671007073313.bam/_temporary/0/task_202510271738013037080282302370339_0308_r_000001
17:38:01.527 INFO SparkHadoopMapRedUtil - attempt_202510271738013037080282302370339_0308_r_000001_0: Committed. Elapsed time: 0 ms.
17:38:01.528 INFO Executor - Finished task 1.0 in stage 75.0 (TID 114). 1729 bytes result sent to driver
17:38:01.528 INFO TaskSetManager - Finished task 1.0 in stage 75.0 (TID 114) in 29 ms on localhost (executor driver) (2/2)
17:38:01.528 INFO TaskSchedulerImpl - Removed TaskSet 75.0, whose tasks have all completed, from pool
17:38:01.529 INFO DAGScheduler - ResultStage 75 (runJob at SparkHadoopWriter.scala:83) finished in 0.039 s
17:38:01.529 INFO DAGScheduler - Job 56 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:01.529 INFO TaskSchedulerImpl - Killing all running tasks in stage 75: Stage finished
17:38:01.529 INFO DAGScheduler - Job 56 finished: runJob at SparkHadoopWriter.scala:83, took 0.123253 s
17:38:01.529 INFO SparkHadoopWriter - Start to commit write Job job_202510271738013037080282302370339_0308.
17:38:01.534 INFO SparkHadoopWriter - Write Job job_202510271738013037080282302370339_0308 committed. Elapsed time: 4 ms.
17:38:01.535 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:01.537 INFO MemoryStore - Block broadcast_138 stored as values in memory (estimated size 297.9 KiB, free 1918.3 MiB)
17:38:01.543 INFO MemoryStore - Block broadcast_138_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1918.2 MiB)
17:38:01.543 INFO BlockManagerInfo - Added broadcast_138_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.7 MiB)
17:38:01.544 INFO SparkContext - Created broadcast 138 from newAPIHadoopFile at PathSplitSource.java:96
17:38:01.566 INFO SparkContext - Starting job: count at ReadsSparkSinkUnitTest.java:222
17:38:01.567 INFO DAGScheduler - Got job 57 (count at ReadsSparkSinkUnitTest.java:222) with 2 output partitions
17:38:01.567 INFO DAGScheduler - Final stage: ResultStage 77 (count at ReadsSparkSinkUnitTest.java:222)
17:38:01.567 INFO DAGScheduler - Parents of final stage: List(ShuffleMapStage 76)
17:38:01.567 INFO DAGScheduler - Missing parents: List()
17:38:01.567 INFO DAGScheduler - Submitting ResultStage 77 (MapPartitionsRDD[299] at repartition at ReadsSparkSinkUnitTest.java:210), which has no missing parents
17:38:01.568 INFO MemoryStore - Block broadcast_139 stored as values in memory (estimated size 6.3 KiB, free 1918.2 MiB)
17:38:01.568 INFO MemoryStore - Block broadcast_139_piece0 stored as bytes in memory (estimated size 3.4 KiB, free 1918.2 MiB)
17:38:01.568 INFO BlockManagerInfo - Added broadcast_139_piece0 in memory on localhost:37953 (size: 3.4 KiB, free: 1919.7 MiB)
17:38:01.569 INFO SparkContext - Created broadcast 139 from broadcast at DAGScheduler.scala:1580
17:38:01.569 INFO DAGScheduler - Submitting 2 missing tasks from ResultStage 77 (MapPartitionsRDD[299] at repartition at ReadsSparkSinkUnitTest.java:210) (first 15 tasks are for partitions Vector(0, 1))
17:38:01.569 INFO TaskSchedulerImpl - Adding task set 77.0 with 2 tasks resource profile 0
17:38:01.570 INFO TaskSetManager - Starting task 0.0 in stage 77.0 (TID 115) (localhost, executor driver, partition 0, NODE_LOCAL, 7860 bytes)
17:38:01.570 INFO TaskSetManager - Starting task 1.0 in stage 77.0 (TID 116) (localhost, executor driver, partition 1, NODE_LOCAL, 7860 bytes)
17:38:01.570 INFO Executor - Running task 0.0 in stage 77.0 (TID 115)
17:38:01.570 INFO Executor - Running task 1.0 in stage 77.0 (TID 116)
17:38:01.572 INFO ShuffleBlockFetcherIterator - Getting 1 (176.4 KiB) non-empty blocks including 1 (176.4 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
17:38:01.572 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
17:38:01.572 INFO ShuffleBlockFetcherIterator - Getting 1 (176.4 KiB) non-empty blocks including 1 (176.4 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
17:38:01.574 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 1 ms
17:38:01.576 INFO Executor - Finished task 0.0 in stage 77.0 (TID 115). 1591 bytes result sent to driver
17:38:01.577 INFO TaskSetManager - Finished task 0.0 in stage 77.0 (TID 115) in 8 ms on localhost (executor driver) (1/2)
17:38:01.577 INFO Executor - Finished task 1.0 in stage 77.0 (TID 116). 1591 bytes result sent to driver
17:38:01.578 INFO TaskSetManager - Finished task 1.0 in stage 77.0 (TID 116) in 8 ms on localhost (executor driver) (2/2)
17:38:01.578 INFO TaskSchedulerImpl - Removed TaskSet 77.0, whose tasks have all completed, from pool
17:38:01.578 INFO DAGScheduler - ResultStage 77 (count at ReadsSparkSinkUnitTest.java:222) finished in 0.011 s
17:38:01.578 INFO DAGScheduler - Job 57 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:01.578 INFO TaskSchedulerImpl - Killing all running tasks in stage 77: Stage finished
17:38:01.578 INFO DAGScheduler - Job 57 finished: count at ReadsSparkSinkUnitTest.java:222, took 0.011881 s
17:38:01.587 INFO FileInputFormat - Total input files to process : 2
17:38:01.590 INFO SparkContext - Starting job: count at ReadsSparkSinkUnitTest.java:222
17:38:01.591 INFO DAGScheduler - Got job 58 (count at ReadsSparkSinkUnitTest.java:222) with 2 output partitions
17:38:01.591 INFO DAGScheduler - Final stage: ResultStage 78 (count at ReadsSparkSinkUnitTest.java:222)
17:38:01.591 INFO DAGScheduler - Parents of final stage: List()
17:38:01.591 INFO DAGScheduler - Missing parents: List()
17:38:01.591 INFO DAGScheduler - Submitting ResultStage 78 (MapPartitionsRDD[315] at filter at ReadsSparkSource.java:96), which has no missing parents
17:38:01.608 INFO MemoryStore - Block broadcast_140 stored as values in memory (estimated size 426.1 KiB, free 1917.8 MiB)
17:38:01.609 INFO MemoryStore - Block broadcast_140_piece0 stored as bytes in memory (estimated size 153.6 KiB, free 1917.6 MiB)
17:38:01.609 INFO BlockManagerInfo - Added broadcast_140_piece0 in memory on localhost:37953 (size: 153.6 KiB, free: 1919.5 MiB)
17:38:01.609 INFO SparkContext - Created broadcast 140 from broadcast at DAGScheduler.scala:1580
17:38:01.610 INFO DAGScheduler - Submitting 2 missing tasks from ResultStage 78 (MapPartitionsRDD[315] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0, 1))
17:38:01.610 INFO TaskSchedulerImpl - Adding task set 78.0 with 2 tasks resource profile 0
17:38:01.610 INFO TaskSetManager - Starting task 0.0 in stage 78.0 (TID 117) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7826 bytes)
17:38:01.611 INFO TaskSetManager - Starting task 1.0 in stage 78.0 (TID 118) (localhost, executor driver, partition 1, PROCESS_LOCAL, 7826 bytes)
17:38:01.611 INFO Executor - Running task 0.0 in stage 78.0 (TID 117)
17:38:01.611 INFO Executor - Running task 1.0 in stage 78.0 (TID 118)
17:38:01.648 INFO NewHadoopRDD - Input split: file:/tmp/ReadsSparkSinkUnitTest12034560671007073313.bam/part-r-00001.bam:0+129979
17:38:01.648 INFO NewHadoopRDD - Input split: file:/tmp/ReadsSparkSinkUnitTest12034560671007073313.bam/part-r-00000.bam:0+133545
17:38:01.650 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:01.650 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:01.651 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:01.651 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:01.658 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:01.658 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:01.661 INFO Executor - Finished task 1.0 in stage 78.0 (TID 118). 989 bytes result sent to driver
17:38:01.661 INFO Executor - Finished task 0.0 in stage 78.0 (TID 117). 989 bytes result sent to driver
17:38:01.661 INFO TaskSetManager - Finished task 1.0 in stage 78.0 (TID 118) in 51 ms on localhost (executor driver) (1/2)
17:38:01.662 INFO TaskSetManager - Finished task 0.0 in stage 78.0 (TID 117) in 52 ms on localhost (executor driver) (2/2)
17:38:01.662 INFO TaskSchedulerImpl - Removed TaskSet 78.0, whose tasks have all completed, from pool
17:38:01.662 INFO DAGScheduler - ResultStage 78 (count at ReadsSparkSinkUnitTest.java:222) finished in 0.071 s
17:38:01.662 INFO DAGScheduler - Job 58 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:01.663 INFO TaskSchedulerImpl - Killing all running tasks in stage 78: Stage finished
17:38:01.663 INFO DAGScheduler - Job 58 finished: count at ReadsSparkSinkUnitTest.java:222, took 0.072306 s
17:38:01.664 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:01.667 INFO MemoryStore - Block broadcast_141 stored as values in memory (estimated size 297.9 KiB, free 1917.3 MiB)
17:38:01.672 INFO MemoryStore - Block broadcast_141_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1917.3 MiB)
17:38:01.673 INFO BlockManagerInfo - Added broadcast_141_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.5 MiB)
17:38:01.673 INFO SparkContext - Created broadcast 141 from newAPIHadoopFile at PathSplitSource.java:96
17:38:01.694 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:01.696 INFO MemoryStore - Block broadcast_142 stored as values in memory (estimated size 297.9 KiB, free 1917.0 MiB)
17:38:01.702 INFO MemoryStore - Block broadcast_142_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1917.0 MiB)
17:38:01.702 INFO BlockManagerInfo - Added broadcast_142_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.4 MiB)
17:38:01.702 INFO SparkContext - Created broadcast 142 from newAPIHadoopFile at PathSplitSource.java:96
17:38:01.722 INFO MemoryStore - Block broadcast_143 stored as values in memory (estimated size 160.7 KiB, free 1916.8 MiB)
17:38:01.728 INFO MemoryStore - Block broadcast_143_piece0 stored as bytes in memory (estimated size 9.6 KiB, free 1916.8 MiB)
17:38:01.728 INFO BlockManagerInfo - Added broadcast_143_piece0 in memory on localhost:37953 (size: 9.6 KiB, free: 1919.4 MiB)
17:38:01.728 INFO SparkContext - Created broadcast 143 from broadcast at ReadsSparkSink.java:133
17:38:01.730 INFO BlockManagerInfo - Removed broadcast_139_piece0 on localhost:37953 in memory (size: 3.4 KiB, free: 1919.4 MiB)
17:38:01.730 INFO MemoryStore - Block broadcast_144 stored as values in memory (estimated size 163.2 KiB, free 1916.6 MiB)
17:38:01.731 INFO BlockManagerInfo - Removed broadcast_137_piece0 on localhost:37953 in memory (size: 56.2 KiB, free: 1919.5 MiB)
17:38:01.731 INFO MemoryStore - Block broadcast_144_piece0 stored as bytes in memory (estimated size 9.6 KiB, free 1916.7 MiB)
17:38:01.731 INFO BlockManagerInfo - Added broadcast_144_piece0 in memory on localhost:37953 (size: 9.6 KiB, free: 1919.5 MiB)
17:38:01.731 INFO SparkContext - Created broadcast 144 from broadcast at AnySamSinkMultiple.java:80
17:38:01.733 INFO BlockManagerInfo - Removed broadcast_140_piece0 on localhost:37953 in memory (size: 153.6 KiB, free: 1919.6 MiB)
17:38:01.733 INFO BlockManagerInfo - Removed broadcast_135_piece0 on localhost:37953 in memory (size: 9.6 KiB, free: 1919.6 MiB)
17:38:01.734 INFO BlockManagerInfo - Removed broadcast_132_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.7 MiB)
17:38:01.734 INFO BlockManagerInfo - Removed broadcast_136_piece0 on localhost:37953 in memory (size: 154.6 KiB, free: 1919.8 MiB)
17:38:01.735 INFO BlockManagerInfo - Removed broadcast_134_piece0 on localhost:37953 in memory (size: 9.6 KiB, free: 1919.8 MiB)
17:38:01.736 INFO BlockManagerInfo - Removed broadcast_138_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.9 MiB)
17:38:01.736 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:01.736 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:01.736 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:01.737 INFO BlockManagerInfo - Removed broadcast_142_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.9 MiB)
17:38:01.745 INFO FileInputFormat - Total input files to process : 1
17:38:01.751 INFO SparkContext - Starting job: runJob at SparkHadoopWriter.scala:83
17:38:01.751 INFO DAGScheduler - Registering RDD 323 (repartition at ReadsSparkSinkUnitTest.java:210) as input to shuffle 17
17:38:01.752 INFO DAGScheduler - Got job 59 (runJob at SparkHadoopWriter.scala:83) with 2 output partitions
17:38:01.752 INFO DAGScheduler - Final stage: ResultStage 80 (runJob at SparkHadoopWriter.scala:83)
17:38:01.752 INFO DAGScheduler - Parents of final stage: List(ShuffleMapStage 79)
17:38:01.752 INFO DAGScheduler - Missing parents: List(ShuffleMapStage 79)
17:38:01.752 INFO DAGScheduler - Submitting ShuffleMapStage 79 (MapPartitionsRDD[323] at repartition at ReadsSparkSinkUnitTest.java:210), which has no missing parents
17:38:01.768 INFO MemoryStore - Block broadcast_145 stored as values in memory (estimated size 427.7 KiB, free 1918.9 MiB)
17:38:01.769 INFO MemoryStore - Block broadcast_145_piece0 stored as bytes in memory (estimated size 154.6 KiB, free 1918.8 MiB)
17:38:01.769 INFO BlockManagerInfo - Added broadcast_145_piece0 in memory on localhost:37953 (size: 154.6 KiB, free: 1919.8 MiB)
17:38:01.769 INFO SparkContext - Created broadcast 145 from broadcast at DAGScheduler.scala:1580
17:38:01.770 INFO DAGScheduler - Submitting 1 missing tasks from ShuffleMapStage 79 (MapPartitionsRDD[323] at repartition at ReadsSparkSinkUnitTest.java:210) (first 15 tasks are for partitions Vector(0))
17:38:01.770 INFO TaskSchedulerImpl - Adding task set 79.0 with 1 tasks resource profile 0
17:38:01.771 INFO TaskSetManager - Starting task 0.0 in stage 79.0 (TID 119) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7861 bytes)
17:38:01.771 INFO Executor - Running task 0.0 in stage 79.0 (TID 119)
17:38:01.805 INFO NewHadoopRDD - Input split: file:/home/runner/work/gatk/gatk/src/test/resources/org/broadinstitute/hellbender/tools/BQSR/HiSeq.1mb.1RG.2k_lines.bam:0+222075
17:38:01.806 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:01.807 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:01.814 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:01.826 INFO Executor - Finished task 0.0 in stage 79.0 (TID 119). 1149 bytes result sent to driver
17:38:01.827 INFO TaskSetManager - Finished task 0.0 in stage 79.0 (TID 119) in 57 ms on localhost (executor driver) (1/1)
17:38:01.827 INFO TaskSchedulerImpl - Removed TaskSet 79.0, whose tasks have all completed, from pool
17:38:01.827 INFO DAGScheduler - ShuffleMapStage 79 (repartition at ReadsSparkSinkUnitTest.java:210) finished in 0.075 s
17:38:01.827 INFO DAGScheduler - looking for newly runnable stages
17:38:01.827 INFO DAGScheduler - running: HashSet()
17:38:01.827 INFO DAGScheduler - waiting: HashSet(ResultStage 80)
17:38:01.827 INFO DAGScheduler - failed: HashSet()
17:38:01.827 INFO DAGScheduler - Submitting ResultStage 80 (MapPartitionsRDD[335] at mapToPair at AnySamSinkMultiple.java:89), which has no missing parents
17:38:01.834 INFO MemoryStore - Block broadcast_146 stored as values in memory (estimated size 150.2 KiB, free 1918.6 MiB)
17:38:01.834 INFO MemoryStore - Block broadcast_146_piece0 stored as bytes in memory (estimated size 56.2 KiB, free 1918.6 MiB)
17:38:01.835 INFO BlockManagerInfo - Added broadcast_146_piece0 in memory on localhost:37953 (size: 56.2 KiB, free: 1919.7 MiB)
17:38:01.835 INFO SparkContext - Created broadcast 146 from broadcast at DAGScheduler.scala:1580
17:38:01.835 INFO DAGScheduler - Submitting 2 missing tasks from ResultStage 80 (MapPartitionsRDD[335] at mapToPair at AnySamSinkMultiple.java:89) (first 15 tasks are for partitions Vector(0, 1))
17:38:01.835 INFO TaskSchedulerImpl - Adding task set 80.0 with 2 tasks resource profile 0
17:38:01.836 INFO TaskSetManager - Starting task 0.0 in stage 80.0 (TID 120) (localhost, executor driver, partition 0, NODE_LOCAL, 7860 bytes)
17:38:01.836 INFO TaskSetManager - Starting task 1.0 in stage 80.0 (TID 121) (localhost, executor driver, partition 1, NODE_LOCAL, 7860 bytes)
17:38:01.837 INFO Executor - Running task 0.0 in stage 80.0 (TID 120)
17:38:01.837 INFO Executor - Running task 1.0 in stage 80.0 (TID 121)
17:38:01.842 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:01.843 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:01.843 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:01.843 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:01.843 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:01.843 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:01.843 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:01.843 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:01.843 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:01.843 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:01.843 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:01.843 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:01.849 WARN IntelDeflaterFactory - Intel Deflater not supported, using Java.util.zip.Deflater
17:38:01.851 INFO ShuffleBlockFetcherIterator - Getting 1 (176.4 KiB) non-empty blocks including 1 (176.4 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
17:38:01.851 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
17:38:01.853 WARN IntelDeflaterFactory - Intel Deflater not supported, using Java.util.zip.Deflater
17:38:01.854 INFO ShuffleBlockFetcherIterator - Getting 1 (176.4 KiB) non-empty blocks including 1 (176.4 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
17:38:01.854 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
17:38:01.861 INFO FileOutputCommitter - Saved output of task 'attempt_202510271738011249179161518259386_0335_r_000001_0' to file:/tmp/ReadsSparkSinkUnitTest18032958144094684569.bam/_temporary/0/task_202510271738011249179161518259386_0335_r_000001
17:38:01.861 INFO SparkHadoopMapRedUtil - attempt_202510271738011249179161518259386_0335_r_000001_0: Committed. Elapsed time: 0 ms.
17:38:01.861 INFO Executor - Finished task 1.0 in stage 80.0 (TID 121). 1729 bytes result sent to driver
17:38:01.862 INFO TaskSetManager - Finished task 1.0 in stage 80.0 (TID 121) in 26 ms on localhost (executor driver) (1/2)
17:38:01.863 INFO FileOutputCommitter - Saved output of task 'attempt_202510271738011249179161518259386_0335_r_000000_0' to file:/tmp/ReadsSparkSinkUnitTest18032958144094684569.bam/_temporary/0/task_202510271738011249179161518259386_0335_r_000000
17:38:01.863 INFO SparkHadoopMapRedUtil - attempt_202510271738011249179161518259386_0335_r_000000_0: Committed. Elapsed time: 0 ms.
17:38:01.863 INFO Executor - Finished task 0.0 in stage 80.0 (TID 120). 1729 bytes result sent to driver
17:38:01.864 INFO TaskSetManager - Finished task 0.0 in stage 80.0 (TID 120) in 28 ms on localhost (executor driver) (2/2)
17:38:01.864 INFO TaskSchedulerImpl - Removed TaskSet 80.0, whose tasks have all completed, from pool
17:38:01.864 INFO DAGScheduler - ResultStage 80 (runJob at SparkHadoopWriter.scala:83) finished in 0.036 s
17:38:01.864 INFO DAGScheduler - Job 59 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:01.864 INFO TaskSchedulerImpl - Killing all running tasks in stage 80: Stage finished
17:38:01.864 INFO DAGScheduler - Job 59 finished: runJob at SparkHadoopWriter.scala:83, took 0.113236 s
17:38:01.865 INFO SparkHadoopWriter - Start to commit write Job job_202510271738011249179161518259386_0335.
17:38:01.869 INFO SparkHadoopWriter - Write Job job_202510271738011249179161518259386_0335 committed. Elapsed time: 4 ms.
17:38:01.870 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:01.872 INFO MemoryStore - Block broadcast_147 stored as values in memory (estimated size 297.9 KiB, free 1918.3 MiB)
17:38:01.878 INFO MemoryStore - Block broadcast_147_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1918.2 MiB)
17:38:01.878 INFO BlockManagerInfo - Added broadcast_147_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.7 MiB)
17:38:01.878 INFO SparkContext - Created broadcast 147 from newAPIHadoopFile at PathSplitSource.java:96
17:38:01.901 INFO SparkContext - Starting job: count at ReadsSparkSinkUnitTest.java:222
17:38:01.901 INFO DAGScheduler - Got job 60 (count at ReadsSparkSinkUnitTest.java:222) with 2 output partitions
17:38:01.901 INFO DAGScheduler - Final stage: ResultStage 82 (count at ReadsSparkSinkUnitTest.java:222)
17:38:01.901 INFO DAGScheduler - Parents of final stage: List(ShuffleMapStage 81)
17:38:01.902 INFO DAGScheduler - Missing parents: List()
17:38:01.902 INFO DAGScheduler - Submitting ResultStage 82 (MapPartitionsRDD[326] at repartition at ReadsSparkSinkUnitTest.java:210), which has no missing parents
17:38:01.903 INFO MemoryStore - Block broadcast_148 stored as values in memory (estimated size 6.3 KiB, free 1918.2 MiB)
17:38:01.903 INFO MemoryStore - Block broadcast_148_piece0 stored as bytes in memory (estimated size 3.4 KiB, free 1918.2 MiB)
17:38:01.903 INFO BlockManagerInfo - Added broadcast_148_piece0 in memory on localhost:37953 (size: 3.4 KiB, free: 1919.7 MiB)
17:38:01.903 INFO SparkContext - Created broadcast 148 from broadcast at DAGScheduler.scala:1580
17:38:01.904 INFO DAGScheduler - Submitting 2 missing tasks from ResultStage 82 (MapPartitionsRDD[326] at repartition at ReadsSparkSinkUnitTest.java:210) (first 15 tasks are for partitions Vector(0, 1))
17:38:01.904 INFO TaskSchedulerImpl - Adding task set 82.0 with 2 tasks resource profile 0
17:38:01.905 INFO TaskSetManager - Starting task 0.0 in stage 82.0 (TID 122) (localhost, executor driver, partition 0, NODE_LOCAL, 7860 bytes)
17:38:01.905 INFO TaskSetManager - Starting task 1.0 in stage 82.0 (TID 123) (localhost, executor driver, partition 1, NODE_LOCAL, 7860 bytes)
17:38:01.905 INFO Executor - Running task 0.0 in stage 82.0 (TID 122)
17:38:01.905 INFO Executor - Running task 1.0 in stage 82.0 (TID 123)
17:38:01.907 INFO ShuffleBlockFetcherIterator - Getting 1 (176.4 KiB) non-empty blocks including 1 (176.4 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
17:38:01.907 INFO ShuffleBlockFetcherIterator - Getting 1 (176.4 KiB) non-empty blocks including 1 (176.4 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
17:38:01.907 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
17:38:01.907 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
17:38:01.911 INFO Executor - Finished task 0.0 in stage 82.0 (TID 122). 1634 bytes result sent to driver
17:38:01.911 INFO Executor - Finished task 1.0 in stage 82.0 (TID 123). 1634 bytes result sent to driver
17:38:01.911 INFO TaskSetManager - Finished task 0.0 in stage 82.0 (TID 122) in 7 ms on localhost (executor driver) (1/2)
17:38:01.912 INFO TaskSetManager - Finished task 1.0 in stage 82.0 (TID 123) in 7 ms on localhost (executor driver) (2/2)
17:38:01.912 INFO TaskSchedulerImpl - Removed TaskSet 82.0, whose tasks have all completed, from pool
17:38:01.912 INFO DAGScheduler - ResultStage 82 (count at ReadsSparkSinkUnitTest.java:222) finished in 0.010 s
17:38:01.912 INFO DAGScheduler - Job 60 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:01.912 INFO TaskSchedulerImpl - Killing all running tasks in stage 82: Stage finished
17:38:01.912 INFO DAGScheduler - Job 60 finished: count at ReadsSparkSinkUnitTest.java:222, took 0.011255 s
17:38:01.921 INFO FileInputFormat - Total input files to process : 2
17:38:01.924 INFO SparkContext - Starting job: count at ReadsSparkSinkUnitTest.java:222
17:38:01.925 INFO DAGScheduler - Got job 61 (count at ReadsSparkSinkUnitTest.java:222) with 2 output partitions
17:38:01.925 INFO DAGScheduler - Final stage: ResultStage 83 (count at ReadsSparkSinkUnitTest.java:222)
17:38:01.925 INFO DAGScheduler - Parents of final stage: List()
17:38:01.925 INFO DAGScheduler - Missing parents: List()
17:38:01.925 INFO DAGScheduler - Submitting ResultStage 83 (MapPartitionsRDD[342] at filter at ReadsSparkSource.java:96), which has no missing parents
17:38:01.941 INFO MemoryStore - Block broadcast_149 stored as values in memory (estimated size 426.1 KiB, free 1917.8 MiB)
17:38:01.942 INFO MemoryStore - Block broadcast_149_piece0 stored as bytes in memory (estimated size 153.6 KiB, free 1917.6 MiB)
17:38:01.942 INFO BlockManagerInfo - Added broadcast_149_piece0 in memory on localhost:37953 (size: 153.6 KiB, free: 1919.5 MiB)
17:38:01.942 INFO SparkContext - Created broadcast 149 from broadcast at DAGScheduler.scala:1580
17:38:01.942 INFO DAGScheduler - Submitting 2 missing tasks from ResultStage 83 (MapPartitionsRDD[342] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0, 1))
17:38:01.942 INFO TaskSchedulerImpl - Adding task set 83.0 with 2 tasks resource profile 0
17:38:01.943 INFO TaskSetManager - Starting task 0.0 in stage 83.0 (TID 124) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7826 bytes)
17:38:01.943 INFO TaskSetManager - Starting task 1.0 in stage 83.0 (TID 125) (localhost, executor driver, partition 1, PROCESS_LOCAL, 7826 bytes)
17:38:01.944 INFO Executor - Running task 0.0 in stage 83.0 (TID 124)
17:38:01.944 INFO Executor - Running task 1.0 in stage 83.0 (TID 125)
17:38:01.981 INFO NewHadoopRDD - Input split: file:/tmp/ReadsSparkSinkUnitTest18032958144094684569.bam/part-r-00000.bam:0+133545
17:38:01.981 INFO NewHadoopRDD - Input split: file:/tmp/ReadsSparkSinkUnitTest18032958144094684569.bam/part-r-00001.bam:0+129979
17:38:01.982 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:01.982 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:01.984 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:01.984 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:01.991 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:01.991 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:01.994 INFO Executor - Finished task 1.0 in stage 83.0 (TID 125). 989 bytes result sent to driver
17:38:01.994 INFO Executor - Finished task 0.0 in stage 83.0 (TID 124). 989 bytes result sent to driver
17:38:01.995 INFO TaskSetManager - Finished task 0.0 in stage 83.0 (TID 124) in 52 ms on localhost (executor driver) (1/2)
17:38:01.995 INFO TaskSetManager - Finished task 1.0 in stage 83.0 (TID 125) in 52 ms on localhost (executor driver) (2/2)
17:38:01.995 INFO TaskSchedulerImpl - Removed TaskSet 83.0, whose tasks have all completed, from pool
17:38:01.995 INFO DAGScheduler - ResultStage 83 (count at ReadsSparkSinkUnitTest.java:222) finished in 0.070 s
17:38:01.995 INFO DAGScheduler - Job 61 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:01.995 INFO TaskSchedulerImpl - Killing all running tasks in stage 83: Stage finished
17:38:01.995 INFO DAGScheduler - Job 61 finished: count at ReadsSparkSinkUnitTest.java:222, took 0.070970 s
17:38:01.996 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:02.001 INFO MemoryStore - Block broadcast_150 stored as values in memory (estimated size 297.9 KiB, free 1917.3 MiB)
17:38:02.013 INFO BlockManagerInfo - Removed broadcast_143_piece0 on localhost:37953 in memory (size: 9.6 KiB, free: 1919.5 MiB)
17:38:02.014 INFO MemoryStore - Block broadcast_150_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1917.5 MiB)
17:38:02.014 INFO BlockManagerInfo - Added broadcast_150_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.5 MiB)
17:38:02.014 INFO BlockManagerInfo - Removed broadcast_149_piece0 on localhost:37953 in memory (size: 153.6 KiB, free: 1919.6 MiB)
17:38:02.015 INFO SparkContext - Created broadcast 150 from newAPIHadoopFile at PathSplitSource.java:96
17:38:02.018 INFO BlockManagerInfo - Removed broadcast_148_piece0 on localhost:37953 in memory (size: 3.4 KiB, free: 1919.6 MiB)
17:38:02.018 INFO BlockManagerInfo - Removed broadcast_144_piece0 on localhost:37953 in memory (size: 9.6 KiB, free: 1919.6 MiB)
17:38:02.019 INFO BlockManagerInfo - Removed broadcast_146_piece0 on localhost:37953 in memory (size: 56.2 KiB, free: 1919.7 MiB)
17:38:02.020 INFO BlockManagerInfo - Removed broadcast_147_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.8 MiB)
17:38:02.021 INFO BlockManagerInfo - Removed broadcast_145_piece0 on localhost:37953 in memory (size: 154.6 KiB, free: 1919.9 MiB)
17:38:02.022 INFO BlockManagerInfo - Removed broadcast_141_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1920.0 MiB)
17:38:02.038 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:02.040 INFO MemoryStore - Block broadcast_151 stored as values in memory (estimated size 297.9 KiB, free 1919.4 MiB)
17:38:02.046 INFO MemoryStore - Block broadcast_151_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1919.3 MiB)
17:38:02.046 INFO BlockManagerInfo - Added broadcast_151_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.9 MiB)
17:38:02.046 INFO SparkContext - Created broadcast 151 from newAPIHadoopFile at PathSplitSource.java:96
17:38:02.066 INFO MemoryStore - Block broadcast_152 stored as values in memory (estimated size 160.7 KiB, free 1919.2 MiB)
17:38:02.067 INFO MemoryStore - Block broadcast_152_piece0 stored as bytes in memory (estimated size 9.6 KiB, free 1919.2 MiB)
17:38:02.067 INFO BlockManagerInfo - Added broadcast_152_piece0 in memory on localhost:37953 (size: 9.6 KiB, free: 1919.9 MiB)
17:38:02.067 INFO SparkContext - Created broadcast 152 from broadcast at ReadsSparkSink.java:133
17:38:02.069 INFO MemoryStore - Block broadcast_153 stored as values in memory (estimated size 163.2 KiB, free 1919.0 MiB)
17:38:02.070 INFO MemoryStore - Block broadcast_153_piece0 stored as bytes in memory (estimated size 9.6 KiB, free 1919.0 MiB)
17:38:02.070 INFO BlockManagerInfo - Added broadcast_153_piece0 in memory on localhost:37953 (size: 9.6 KiB, free: 1919.9 MiB)
17:38:02.070 INFO SparkContext - Created broadcast 153 from broadcast at AnySamSinkMultiple.java:80
17:38:02.072 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:02.072 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:02.072 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:02.081 INFO FileInputFormat - Total input files to process : 1
17:38:02.087 INFO SparkContext - Starting job: runJob at SparkHadoopWriter.scala:83
17:38:02.088 INFO DAGScheduler - Registering RDD 350 (repartition at ReadsSparkSinkUnitTest.java:210) as input to shuffle 18
17:38:02.088 INFO DAGScheduler - Got job 62 (runJob at SparkHadoopWriter.scala:83) with 2 output partitions
17:38:02.088 INFO DAGScheduler - Final stage: ResultStage 85 (runJob at SparkHadoopWriter.scala:83)
17:38:02.088 INFO DAGScheduler - Parents of final stage: List(ShuffleMapStage 84)
17:38:02.088 INFO DAGScheduler - Missing parents: List(ShuffleMapStage 84)
17:38:02.088 INFO DAGScheduler - Submitting ShuffleMapStage 84 (MapPartitionsRDD[350] at repartition at ReadsSparkSinkUnitTest.java:210), which has no missing parents
17:38:02.104 INFO MemoryStore - Block broadcast_154 stored as values in memory (estimated size 427.7 KiB, free 1918.6 MiB)
17:38:02.105 INFO MemoryStore - Block broadcast_154_piece0 stored as bytes in memory (estimated size 154.6 KiB, free 1918.4 MiB)
17:38:02.105 INFO BlockManagerInfo - Added broadcast_154_piece0 in memory on localhost:37953 (size: 154.6 KiB, free: 1919.7 MiB)
17:38:02.106 INFO SparkContext - Created broadcast 154 from broadcast at DAGScheduler.scala:1580
17:38:02.106 INFO DAGScheduler - Submitting 1 missing tasks from ShuffleMapStage 84 (MapPartitionsRDD[350] at repartition at ReadsSparkSinkUnitTest.java:210) (first 15 tasks are for partitions Vector(0))
17:38:02.106 INFO TaskSchedulerImpl - Adding task set 84.0 with 1 tasks resource profile 0
17:38:02.107 INFO TaskSetManager - Starting task 0.0 in stage 84.0 (TID 126) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7861 bytes)
17:38:02.107 INFO Executor - Running task 0.0 in stage 84.0 (TID 126)
17:38:02.141 INFO NewHadoopRDD - Input split: file:/home/runner/work/gatk/gatk/src/test/resources/org/broadinstitute/hellbender/tools/BQSR/HiSeq.1mb.1RG.2k_lines.bam:0+222075
17:38:02.142 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:02.144 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:02.151 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:02.163 INFO Executor - Finished task 0.0 in stage 84.0 (TID 126). 1149 bytes result sent to driver
17:38:02.164 INFO TaskSetManager - Finished task 0.0 in stage 84.0 (TID 126) in 57 ms on localhost (executor driver) (1/1)
17:38:02.164 INFO TaskSchedulerImpl - Removed TaskSet 84.0, whose tasks have all completed, from pool
17:38:02.164 INFO DAGScheduler - ShuffleMapStage 84 (repartition at ReadsSparkSinkUnitTest.java:210) finished in 0.076 s
17:38:02.164 INFO DAGScheduler - looking for newly runnable stages
17:38:02.164 INFO DAGScheduler - running: HashSet()
17:38:02.164 INFO DAGScheduler - waiting: HashSet(ResultStage 85)
17:38:02.164 INFO DAGScheduler - failed: HashSet()
17:38:02.164 INFO DAGScheduler - Submitting ResultStage 85 (MapPartitionsRDD[362] at mapToPair at AnySamSinkMultiple.java:89), which has no missing parents
17:38:02.170 INFO MemoryStore - Block broadcast_155 stored as values in memory (estimated size 150.2 KiB, free 1918.3 MiB)
17:38:02.171 INFO MemoryStore - Block broadcast_155_piece0 stored as bytes in memory (estimated size 56.3 KiB, free 1918.2 MiB)
17:38:02.171 INFO BlockManagerInfo - Added broadcast_155_piece0 in memory on localhost:37953 (size: 56.3 KiB, free: 1919.7 MiB)
17:38:02.172 INFO SparkContext - Created broadcast 155 from broadcast at DAGScheduler.scala:1580
17:38:02.172 INFO DAGScheduler - Submitting 2 missing tasks from ResultStage 85 (MapPartitionsRDD[362] at mapToPair at AnySamSinkMultiple.java:89) (first 15 tasks are for partitions Vector(0, 1))
17:38:02.172 INFO TaskSchedulerImpl - Adding task set 85.0 with 2 tasks resource profile 0
17:38:02.173 INFO TaskSetManager - Starting task 0.0 in stage 85.0 (TID 127) (localhost, executor driver, partition 0, NODE_LOCAL, 7860 bytes)
17:38:02.173 INFO TaskSetManager - Starting task 1.0 in stage 85.0 (TID 128) (localhost, executor driver, partition 1, NODE_LOCAL, 7860 bytes)
17:38:02.173 INFO Executor - Running task 1.0 in stage 85.0 (TID 128)
17:38:02.173 INFO Executor - Running task 0.0 in stage 85.0 (TID 127)
17:38:02.179 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:02.179 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:02.179 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:02.179 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:02.179 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:02.179 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:02.179 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:02.179 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:02.179 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:02.180 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:02.180 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:02.180 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:02.186 WARN IntelDeflaterFactory - Intel Deflater not supported, using Java.util.zip.Deflater
17:38:02.187 INFO ShuffleBlockFetcherIterator - Getting 1 (176.4 KiB) non-empty blocks including 1 (176.4 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
17:38:02.187 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
17:38:02.188 WARN IntelDeflaterFactory - Intel Deflater not supported, using Java.util.zip.Deflater
17:38:02.189 INFO ShuffleBlockFetcherIterator - Getting 1 (176.4 KiB) non-empty blocks including 1 (176.4 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
17:38:02.189 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
17:38:02.196 INFO FileOutputCommitter - Saved output of task 'attempt_202510271738025439411991095351684_0362_r_000000_0' to file:/tmp/ReadsSparkSinkUnitTest13631781286460174334.bam/_temporary/0/task_202510271738025439411991095351684_0362_r_000000
17:38:02.196 INFO SparkHadoopMapRedUtil - attempt_202510271738025439411991095351684_0362_r_000000_0: Committed. Elapsed time: 0 ms.
17:38:02.197 INFO Executor - Finished task 0.0 in stage 85.0 (TID 127). 1729 bytes result sent to driver
17:38:02.198 INFO FileOutputCommitter - Saved output of task 'attempt_202510271738025439411991095351684_0362_r_000001_0' to file:/tmp/ReadsSparkSinkUnitTest13631781286460174334.bam/_temporary/0/task_202510271738025439411991095351684_0362_r_000001
17:38:02.198 INFO SparkHadoopMapRedUtil - attempt_202510271738025439411991095351684_0362_r_000001_0: Committed. Elapsed time: 0 ms.
17:38:02.198 INFO TaskSetManager - Finished task 0.0 in stage 85.0 (TID 127) in 26 ms on localhost (executor driver) (1/2)
17:38:02.199 INFO Executor - Finished task 1.0 in stage 85.0 (TID 128). 1729 bytes result sent to driver
17:38:02.199 INFO TaskSetManager - Finished task 1.0 in stage 85.0 (TID 128) in 26 ms on localhost (executor driver) (2/2)
17:38:02.199 INFO TaskSchedulerImpl - Removed TaskSet 85.0, whose tasks have all completed, from pool
17:38:02.199 INFO DAGScheduler - ResultStage 85 (runJob at SparkHadoopWriter.scala:83) finished in 0.034 s
17:38:02.199 INFO DAGScheduler - Job 62 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:02.199 INFO TaskSchedulerImpl - Killing all running tasks in stage 85: Stage finished
17:38:02.200 INFO DAGScheduler - Job 62 finished: runJob at SparkHadoopWriter.scala:83, took 0.112564 s
17:38:02.200 INFO SparkHadoopWriter - Start to commit write Job job_202510271738025439411991095351684_0362.
17:38:02.204 INFO SparkHadoopWriter - Write Job job_202510271738025439411991095351684_0362 committed. Elapsed time: 4 ms.
17:38:02.205 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:02.208 INFO MemoryStore - Block broadcast_156 stored as values in memory (estimated size 297.9 KiB, free 1917.9 MiB)
17:38:02.214 INFO MemoryStore - Block broadcast_156_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1917.9 MiB)
17:38:02.214 INFO BlockManagerInfo - Added broadcast_156_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.6 MiB)
17:38:02.214 INFO SparkContext - Created broadcast 156 from newAPIHadoopFile at PathSplitSource.java:96
17:38:02.237 INFO SparkContext - Starting job: count at ReadsSparkSinkUnitTest.java:222
17:38:02.237 INFO DAGScheduler - Got job 63 (count at ReadsSparkSinkUnitTest.java:222) with 2 output partitions
17:38:02.237 INFO DAGScheduler - Final stage: ResultStage 87 (count at ReadsSparkSinkUnitTest.java:222)
17:38:02.237 INFO DAGScheduler - Parents of final stage: List(ShuffleMapStage 86)
17:38:02.237 INFO DAGScheduler - Missing parents: List()
17:38:02.237 INFO DAGScheduler - Submitting ResultStage 87 (MapPartitionsRDD[353] at repartition at ReadsSparkSinkUnitTest.java:210), which has no missing parents
17:38:02.238 INFO MemoryStore - Block broadcast_157 stored as values in memory (estimated size 6.3 KiB, free 1917.9 MiB)
17:38:02.239 INFO MemoryStore - Block broadcast_157_piece0 stored as bytes in memory (estimated size 3.4 KiB, free 1917.9 MiB)
17:38:02.239 INFO BlockManagerInfo - Added broadcast_157_piece0 in memory on localhost:37953 (size: 3.4 KiB, free: 1919.6 MiB)
17:38:02.239 INFO SparkContext - Created broadcast 157 from broadcast at DAGScheduler.scala:1580
17:38:02.239 INFO DAGScheduler - Submitting 2 missing tasks from ResultStage 87 (MapPartitionsRDD[353] at repartition at ReadsSparkSinkUnitTest.java:210) (first 15 tasks are for partitions Vector(0, 1))
17:38:02.239 INFO TaskSchedulerImpl - Adding task set 87.0 with 2 tasks resource profile 0
17:38:02.241 INFO TaskSetManager - Starting task 0.0 in stage 87.0 (TID 129) (localhost, executor driver, partition 0, NODE_LOCAL, 7860 bytes)
17:38:02.241 INFO TaskSetManager - Starting task 1.0 in stage 87.0 (TID 130) (localhost, executor driver, partition 1, NODE_LOCAL, 7860 bytes)
17:38:02.241 INFO Executor - Running task 0.0 in stage 87.0 (TID 129)
17:38:02.241 INFO Executor - Running task 1.0 in stage 87.0 (TID 130)
17:38:02.243 INFO ShuffleBlockFetcherIterator - Getting 1 (176.4 KiB) non-empty blocks including 1 (176.4 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
17:38:02.243 INFO ShuffleBlockFetcherIterator - Getting 1 (176.4 KiB) non-empty blocks including 1 (176.4 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
17:38:02.243 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
17:38:02.243 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
17:38:02.252 INFO Executor - Finished task 1.0 in stage 87.0 (TID 130). 1720 bytes result sent to driver
17:38:02.253 INFO BlockManagerInfo - Removed broadcast_152_piece0 on localhost:37953 in memory (size: 9.6 KiB, free: 1919.6 MiB)
17:38:02.253 INFO Executor - Finished task 0.0 in stage 87.0 (TID 129). 1720 bytes result sent to driver
17:38:02.253 INFO TaskSetManager - Finished task 1.0 in stage 87.0 (TID 130) in 12 ms on localhost (executor driver) (1/2)
17:38:02.253 INFO TaskSetManager - Finished task 0.0 in stage 87.0 (TID 129) in 13 ms on localhost (executor driver) (2/2)
17:38:02.253 INFO TaskSchedulerImpl - Removed TaskSet 87.0, whose tasks have all completed, from pool
17:38:02.254 INFO DAGScheduler - ResultStage 87 (count at ReadsSparkSinkUnitTest.java:222) finished in 0.016 s
17:38:02.254 INFO DAGScheduler - Job 63 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:02.254 INFO TaskSchedulerImpl - Killing all running tasks in stage 87: Stage finished
17:38:02.254 INFO DAGScheduler - Job 63 finished: count at ReadsSparkSinkUnitTest.java:222, took 0.017550 s
17:38:02.254 INFO BlockManagerInfo - Removed broadcast_151_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.7 MiB)
17:38:02.256 INFO BlockManagerInfo - Removed broadcast_155_piece0 on localhost:37953 in memory (size: 56.3 KiB, free: 1919.7 MiB)
17:38:02.257 INFO BlockManagerInfo - Removed broadcast_153_piece0 on localhost:37953 in memory (size: 9.6 KiB, free: 1919.7 MiB)
17:38:02.258 INFO BlockManagerInfo - Removed broadcast_154_piece0 on localhost:37953 in memory (size: 154.6 KiB, free: 1919.9 MiB)
17:38:02.264 INFO FileInputFormat - Total input files to process : 2
17:38:02.268 INFO SparkContext - Starting job: count at ReadsSparkSinkUnitTest.java:222
17:38:02.268 INFO DAGScheduler - Got job 64 (count at ReadsSparkSinkUnitTest.java:222) with 2 output partitions
17:38:02.268 INFO DAGScheduler - Final stage: ResultStage 88 (count at ReadsSparkSinkUnitTest.java:222)
17:38:02.268 INFO DAGScheduler - Parents of final stage: List()
17:38:02.268 INFO DAGScheduler - Missing parents: List()
17:38:02.268 INFO DAGScheduler - Submitting ResultStage 88 (MapPartitionsRDD[369] at filter at ReadsSparkSource.java:96), which has no missing parents
17:38:02.284 INFO MemoryStore - Block broadcast_158 stored as values in memory (estimated size 426.1 KiB, free 1918.9 MiB)
17:38:02.285 INFO MemoryStore - Block broadcast_158_piece0 stored as bytes in memory (estimated size 153.6 KiB, free 1918.7 MiB)
17:38:02.285 INFO BlockManagerInfo - Added broadcast_158_piece0 in memory on localhost:37953 (size: 153.6 KiB, free: 1919.7 MiB)
17:38:02.286 INFO SparkContext - Created broadcast 158 from broadcast at DAGScheduler.scala:1580
17:38:02.286 INFO DAGScheduler - Submitting 2 missing tasks from ResultStage 88 (MapPartitionsRDD[369] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0, 1))
17:38:02.286 INFO TaskSchedulerImpl - Adding task set 88.0 with 2 tasks resource profile 0
17:38:02.287 INFO TaskSetManager - Starting task 0.0 in stage 88.0 (TID 131) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7826 bytes)
17:38:02.287 INFO TaskSetManager - Starting task 1.0 in stage 88.0 (TID 132) (localhost, executor driver, partition 1, PROCESS_LOCAL, 7826 bytes)
17:38:02.287 INFO Executor - Running task 0.0 in stage 88.0 (TID 131)
17:38:02.287 INFO Executor - Running task 1.0 in stage 88.0 (TID 132)
17:38:02.331 INFO NewHadoopRDD - Input split: file:/tmp/ReadsSparkSinkUnitTest13631781286460174334.bam/part-r-00001.bam:0+129979
17:38:02.331 INFO NewHadoopRDD - Input split: file:/tmp/ReadsSparkSinkUnitTest13631781286460174334.bam/part-r-00000.bam:0+133545
17:38:02.332 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:02.332 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:02.333 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:02.333 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:02.340 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:02.340 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:02.343 INFO Executor - Finished task 1.0 in stage 88.0 (TID 132). 989 bytes result sent to driver
17:38:02.343 INFO Executor - Finished task 0.0 in stage 88.0 (TID 131). 989 bytes result sent to driver
17:38:02.344 INFO TaskSetManager - Finished task 1.0 in stage 88.0 (TID 132) in 57 ms on localhost (executor driver) (1/2)
17:38:02.345 INFO TaskSetManager - Finished task 0.0 in stage 88.0 (TID 131) in 58 ms on localhost (executor driver) (2/2)
17:38:02.345 INFO TaskSchedulerImpl - Removed TaskSet 88.0, whose tasks have all completed, from pool
17:38:02.345 INFO DAGScheduler - ResultStage 88 (count at ReadsSparkSinkUnitTest.java:222) finished in 0.076 s
17:38:02.345 INFO DAGScheduler - Job 64 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:02.345 INFO TaskSchedulerImpl - Killing all running tasks in stage 88: Stage finished
17:38:02.346 INFO DAGScheduler - Job 64 finished: count at ReadsSparkSinkUnitTest.java:222, took 0.077872 s
17:38:02.347 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:02.350 INFO MemoryStore - Block broadcast_159 stored as values in memory (estimated size 297.9 KiB, free 1918.5 MiB)
17:38:02.356 INFO MemoryStore - Block broadcast_159_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1918.4 MiB)
17:38:02.356 INFO BlockManagerInfo - Added broadcast_159_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.7 MiB)
17:38:02.357 INFO SparkContext - Created broadcast 159 from newAPIHadoopFile at PathSplitSource.java:96
17:38:02.378 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:02.380 INFO MemoryStore - Block broadcast_160 stored as values in memory (estimated size 297.9 KiB, free 1918.1 MiB)
17:38:02.386 INFO MemoryStore - Block broadcast_160_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1918.1 MiB)
17:38:02.386 INFO BlockManagerInfo - Added broadcast_160_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.7 MiB)
17:38:02.386 INFO SparkContext - Created broadcast 160 from newAPIHadoopFile at PathSplitSource.java:96
17:38:02.405 INFO MemoryStore - Block broadcast_161 stored as values in memory (estimated size 160.7 KiB, free 1917.9 MiB)
17:38:02.406 INFO MemoryStore - Block broadcast_161_piece0 stored as bytes in memory (estimated size 9.6 KiB, free 1917.9 MiB)
17:38:02.406 INFO BlockManagerInfo - Added broadcast_161_piece0 in memory on localhost:37953 (size: 9.6 KiB, free: 1919.6 MiB)
17:38:02.407 INFO SparkContext - Created broadcast 161 from broadcast at ReadsSparkSink.java:133
17:38:02.408 INFO MemoryStore - Block broadcast_162 stored as values in memory (estimated size 163.2 KiB, free 1917.7 MiB)
17:38:02.409 INFO MemoryStore - Block broadcast_162_piece0 stored as bytes in memory (estimated size 9.6 KiB, free 1917.7 MiB)
17:38:02.409 INFO BlockManagerInfo - Added broadcast_162_piece0 in memory on localhost:37953 (size: 9.6 KiB, free: 1919.6 MiB)
17:38:02.409 INFO SparkContext - Created broadcast 162 from broadcast at AnySamSinkMultiple.java:80
17:38:02.411 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:02.411 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:02.411 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:02.421 INFO FileInputFormat - Total input files to process : 1
17:38:02.427 INFO SparkContext - Starting job: runJob at SparkHadoopWriter.scala:83
17:38:02.427 INFO DAGScheduler - Registering RDD 377 (repartition at ReadsSparkSinkUnitTest.java:210) as input to shuffle 19
17:38:02.427 INFO DAGScheduler - Got job 65 (runJob at SparkHadoopWriter.scala:83) with 2 output partitions
17:38:02.427 INFO DAGScheduler - Final stage: ResultStage 90 (runJob at SparkHadoopWriter.scala:83)
17:38:02.428 INFO DAGScheduler - Parents of final stage: List(ShuffleMapStage 89)
17:38:02.428 INFO DAGScheduler - Missing parents: List(ShuffleMapStage 89)
17:38:02.428 INFO DAGScheduler - Submitting ShuffleMapStage 89 (MapPartitionsRDD[377] at repartition at ReadsSparkSinkUnitTest.java:210), which has no missing parents
17:38:02.443 INFO MemoryStore - Block broadcast_163 stored as values in memory (estimated size 427.7 KiB, free 1917.3 MiB)
17:38:02.445 INFO MemoryStore - Block broadcast_163_piece0 stored as bytes in memory (estimated size 154.6 KiB, free 1917.2 MiB)
17:38:02.445 INFO BlockManagerInfo - Added broadcast_163_piece0 in memory on localhost:37953 (size: 154.6 KiB, free: 1919.5 MiB)
17:38:02.445 INFO SparkContext - Created broadcast 163 from broadcast at DAGScheduler.scala:1580
17:38:02.445 INFO DAGScheduler - Submitting 1 missing tasks from ShuffleMapStage 89 (MapPartitionsRDD[377] at repartition at ReadsSparkSinkUnitTest.java:210) (first 15 tasks are for partitions Vector(0))
17:38:02.445 INFO TaskSchedulerImpl - Adding task set 89.0 with 1 tasks resource profile 0
17:38:02.446 INFO TaskSetManager - Starting task 0.0 in stage 89.0 (TID 133) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7861 bytes)
17:38:02.446 INFO Executor - Running task 0.0 in stage 89.0 (TID 133)
17:38:02.479 INFO NewHadoopRDD - Input split: file:/home/runner/work/gatk/gatk/src/test/resources/org/broadinstitute/hellbender/tools/BQSR/HiSeq.1mb.1RG.2k_lines.bam:0+222075
17:38:02.481 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:02.482 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:02.489 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:02.501 INFO Executor - Finished task 0.0 in stage 89.0 (TID 133). 1149 bytes result sent to driver
17:38:02.501 INFO TaskSetManager - Finished task 0.0 in stage 89.0 (TID 133) in 55 ms on localhost (executor driver) (1/1)
17:38:02.501 INFO TaskSchedulerImpl - Removed TaskSet 89.0, whose tasks have all completed, from pool
17:38:02.501 INFO DAGScheduler - ShuffleMapStage 89 (repartition at ReadsSparkSinkUnitTest.java:210) finished in 0.073 s
17:38:02.501 INFO DAGScheduler - looking for newly runnable stages
17:38:02.501 INFO DAGScheduler - running: HashSet()
17:38:02.501 INFO DAGScheduler - waiting: HashSet(ResultStage 90)
17:38:02.501 INFO DAGScheduler - failed: HashSet()
17:38:02.502 INFO DAGScheduler - Submitting ResultStage 90 (MapPartitionsRDD[389] at mapToPair at AnySamSinkMultiple.java:89), which has no missing parents
17:38:02.508 INFO MemoryStore - Block broadcast_164 stored as values in memory (estimated size 150.2 KiB, free 1917.0 MiB)
17:38:02.509 INFO MemoryStore - Block broadcast_164_piece0 stored as bytes in memory (estimated size 56.2 KiB, free 1917.0 MiB)
17:38:02.509 INFO BlockManagerInfo - Added broadcast_164_piece0 in memory on localhost:37953 (size: 56.2 KiB, free: 1919.4 MiB)
17:38:02.509 INFO SparkContext - Created broadcast 164 from broadcast at DAGScheduler.scala:1580
17:38:02.509 INFO DAGScheduler - Submitting 2 missing tasks from ResultStage 90 (MapPartitionsRDD[389] at mapToPair at AnySamSinkMultiple.java:89) (first 15 tasks are for partitions Vector(0, 1))
17:38:02.509 INFO TaskSchedulerImpl - Adding task set 90.0 with 2 tasks resource profile 0
17:38:02.510 INFO TaskSetManager - Starting task 0.0 in stage 90.0 (TID 134) (localhost, executor driver, partition 0, NODE_LOCAL, 7860 bytes)
17:38:02.510 INFO TaskSetManager - Starting task 1.0 in stage 90.0 (TID 135) (localhost, executor driver, partition 1, NODE_LOCAL, 7860 bytes)
17:38:02.511 INFO Executor - Running task 0.0 in stage 90.0 (TID 134)
17:38:02.511 INFO Executor - Running task 1.0 in stage 90.0 (TID 135)
17:38:02.516 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:02.516 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:02.516 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:02.517 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:02.517 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:02.517 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:02.517 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:02.517 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:02.517 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:02.517 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:02.517 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:02.517 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:02.523 WARN IntelDeflaterFactory - Intel Deflater not supported, using Java.util.zip.Deflater
17:38:02.524 INFO ShuffleBlockFetcherIterator - Getting 1 (176.4 KiB) non-empty blocks including 1 (176.4 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
17:38:02.524 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
17:38:02.526 WARN IntelDeflaterFactory - Intel Deflater not supported, using Java.util.zip.Deflater
17:38:02.527 INFO ShuffleBlockFetcherIterator - Getting 1 (176.4 KiB) non-empty blocks including 1 (176.4 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
17:38:02.527 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
17:38:02.533 INFO FileOutputCommitter - Saved output of task 'attempt_202510271738021132720676143695834_0389_r_000000_0' to file:/tmp/ReadsSparkSinkUnitTest19656038191735390335.bam/_temporary/0/task_202510271738021132720676143695834_0389_r_000000
17:38:02.533 INFO SparkHadoopMapRedUtil - attempt_202510271738021132720676143695834_0389_r_000000_0: Committed. Elapsed time: 0 ms.
17:38:02.541 INFO Executor - Finished task 0.0 in stage 90.0 (TID 134). 1815 bytes result sent to driver
17:38:02.542 INFO TaskSetManager - Finished task 0.0 in stage 90.0 (TID 134) in 32 ms on localhost (executor driver) (1/2)
17:38:02.542 INFO BlockManagerInfo - Removed broadcast_150_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.5 MiB)
17:38:02.542 INFO FileOutputCommitter - Saved output of task 'attempt_202510271738021132720676143695834_0389_r_000001_0' to file:/tmp/ReadsSparkSinkUnitTest19656038191735390335.bam/_temporary/0/task_202510271738021132720676143695834_0389_r_000001
17:38:02.542 INFO SparkHadoopMapRedUtil - attempt_202510271738021132720676143695834_0389_r_000001_0: Committed. Elapsed time: 0 ms.
17:38:02.543 INFO Executor - Finished task 1.0 in stage 90.0 (TID 135). 1772 bytes result sent to driver
17:38:02.543 INFO TaskSetManager - Finished task 1.0 in stage 90.0 (TID 135) in 33 ms on localhost (executor driver) (2/2)
17:38:02.543 INFO TaskSchedulerImpl - Removed TaskSet 90.0, whose tasks have all completed, from pool
17:38:02.543 INFO DAGScheduler - ResultStage 90 (runJob at SparkHadoopWriter.scala:83) finished in 0.041 s
17:38:02.544 INFO DAGScheduler - Job 65 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:02.544 INFO TaskSchedulerImpl - Killing all running tasks in stage 90: Stage finished
17:38:02.544 INFO BlockManagerInfo - Removed broadcast_160_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.5 MiB)
17:38:02.544 INFO DAGScheduler - Job 65 finished: runJob at SparkHadoopWriter.scala:83, took 0.117161 s
17:38:02.545 INFO SparkHadoopWriter - Start to commit write Job job_202510271738021132720676143695834_0389.
17:38:02.545 INFO BlockManagerInfo - Removed broadcast_158_piece0 on localhost:37953 in memory (size: 153.6 KiB, free: 1919.7 MiB)
17:38:02.546 INFO BlockManagerInfo - Removed broadcast_157_piece0 on localhost:37953 in memory (size: 3.4 KiB, free: 1919.7 MiB)
17:38:02.546 INFO BlockManagerInfo - Removed broadcast_163_piece0 on localhost:37953 in memory (size: 154.6 KiB, free: 1919.8 MiB)
17:38:02.547 INFO BlockManagerInfo - Removed broadcast_156_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.9 MiB)
17:38:02.549 INFO SparkHadoopWriter - Write Job job_202510271738021132720676143695834_0389 committed. Elapsed time: 4 ms.
17:38:02.550 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:02.552 INFO MemoryStore - Block broadcast_165 stored as values in memory (estimated size 297.9 KiB, free 1918.8 MiB)
17:38:02.557 INFO MemoryStore - Block broadcast_165_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1918.8 MiB)
17:38:02.558 INFO BlockManagerInfo - Added broadcast_165_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.8 MiB)
17:38:02.558 INFO SparkContext - Created broadcast 165 from newAPIHadoopFile at PathSplitSource.java:96
17:38:02.580 INFO SparkContext - Starting job: count at ReadsSparkSinkUnitTest.java:222
17:38:02.580 INFO DAGScheduler - Got job 66 (count at ReadsSparkSinkUnitTest.java:222) with 2 output partitions
17:38:02.580 INFO DAGScheduler - Final stage: ResultStage 92 (count at ReadsSparkSinkUnitTest.java:222)
17:38:02.580 INFO DAGScheduler - Parents of final stage: List(ShuffleMapStage 91)
17:38:02.580 INFO DAGScheduler - Missing parents: List()
17:38:02.581 INFO DAGScheduler - Submitting ResultStage 92 (MapPartitionsRDD[380] at repartition at ReadsSparkSinkUnitTest.java:210), which has no missing parents
17:38:02.581 INFO MemoryStore - Block broadcast_166 stored as values in memory (estimated size 6.3 KiB, free 1918.8 MiB)
17:38:02.582 INFO MemoryStore - Block broadcast_166_piece0 stored as bytes in memory (estimated size 3.4 KiB, free 1918.8 MiB)
17:38:02.582 INFO BlockManagerInfo - Added broadcast_166_piece0 in memory on localhost:37953 (size: 3.4 KiB, free: 1919.8 MiB)
17:38:02.582 INFO SparkContext - Created broadcast 166 from broadcast at DAGScheduler.scala:1580
17:38:02.582 INFO DAGScheduler - Submitting 2 missing tasks from ResultStage 92 (MapPartitionsRDD[380] at repartition at ReadsSparkSinkUnitTest.java:210) (first 15 tasks are for partitions Vector(0, 1))
17:38:02.582 INFO TaskSchedulerImpl - Adding task set 92.0 with 2 tasks resource profile 0
17:38:02.583 INFO TaskSetManager - Starting task 0.0 in stage 92.0 (TID 136) (localhost, executor driver, partition 0, NODE_LOCAL, 7860 bytes)
17:38:02.583 INFO TaskSetManager - Starting task 1.0 in stage 92.0 (TID 137) (localhost, executor driver, partition 1, NODE_LOCAL, 7860 bytes)
17:38:02.584 INFO Executor - Running task 1.0 in stage 92.0 (TID 137)
17:38:02.585 INFO Executor - Running task 0.0 in stage 92.0 (TID 136)
17:38:02.586 INFO ShuffleBlockFetcherIterator - Getting 1 (176.4 KiB) non-empty blocks including 1 (176.4 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
17:38:02.586 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
17:38:02.586 INFO ShuffleBlockFetcherIterator - Getting 1 (176.4 KiB) non-empty blocks including 1 (176.4 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
17:38:02.586 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
17:38:02.590 INFO Executor - Finished task 1.0 in stage 92.0 (TID 137). 1591 bytes result sent to driver
17:38:02.590 INFO Executor - Finished task 0.0 in stage 92.0 (TID 136). 1634 bytes result sent to driver
17:38:02.594 INFO TaskSetManager - Finished task 0.0 in stage 92.0 (TID 136) in 11 ms on localhost (executor driver) (1/2)
17:38:02.594 INFO TaskSetManager - Finished task 1.0 in stage 92.0 (TID 137) in 11 ms on localhost (executor driver) (2/2)
17:38:02.594 INFO TaskSchedulerImpl - Removed TaskSet 92.0, whose tasks have all completed, from pool
17:38:02.594 INFO DAGScheduler - ResultStage 92 (count at ReadsSparkSinkUnitTest.java:222) finished in 0.013 s
17:38:02.594 INFO DAGScheduler - Job 66 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:02.594 INFO TaskSchedulerImpl - Killing all running tasks in stage 92: Stage finished
17:38:02.594 INFO DAGScheduler - Job 66 finished: count at ReadsSparkSinkUnitTest.java:222, took 0.014383 s
17:38:02.603 INFO FileInputFormat - Total input files to process : 2
17:38:02.607 INFO SparkContext - Starting job: count at ReadsSparkSinkUnitTest.java:222
17:38:02.607 INFO DAGScheduler - Got job 67 (count at ReadsSparkSinkUnitTest.java:222) with 2 output partitions
17:38:02.607 INFO DAGScheduler - Final stage: ResultStage 93 (count at ReadsSparkSinkUnitTest.java:222)
17:38:02.607 INFO DAGScheduler - Parents of final stage: List()
17:38:02.607 INFO DAGScheduler - Missing parents: List()
17:38:02.608 INFO DAGScheduler - Submitting ResultStage 93 (MapPartitionsRDD[396] at filter at ReadsSparkSource.java:96), which has no missing parents
17:38:02.624 INFO MemoryStore - Block broadcast_167 stored as values in memory (estimated size 426.1 KiB, free 1918.4 MiB)
17:38:02.625 INFO MemoryStore - Block broadcast_167_piece0 stored as bytes in memory (estimated size 153.6 KiB, free 1918.2 MiB)
17:38:02.625 INFO BlockManagerInfo - Added broadcast_167_piece0 in memory on localhost:37953 (size: 153.6 KiB, free: 1919.7 MiB)
17:38:02.625 INFO SparkContext - Created broadcast 167 from broadcast at DAGScheduler.scala:1580
17:38:02.625 INFO DAGScheduler - Submitting 2 missing tasks from ResultStage 93 (MapPartitionsRDD[396] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0, 1))
17:38:02.625 INFO TaskSchedulerImpl - Adding task set 93.0 with 2 tasks resource profile 0
17:38:02.626 INFO TaskSetManager - Starting task 0.0 in stage 93.0 (TID 138) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7826 bytes)
17:38:02.626 INFO TaskSetManager - Starting task 1.0 in stage 93.0 (TID 139) (localhost, executor driver, partition 1, PROCESS_LOCAL, 7826 bytes)
17:38:02.627 INFO Executor - Running task 1.0 in stage 93.0 (TID 139)
17:38:02.627 INFO Executor - Running task 0.0 in stage 93.0 (TID 138)
17:38:02.674 INFO NewHadoopRDD - Input split: file:/tmp/ReadsSparkSinkUnitTest19656038191735390335.bam/part-r-00001.bam:0+129979
17:38:02.675 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:02.676 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:02.677 INFO NewHadoopRDD - Input split: file:/tmp/ReadsSparkSinkUnitTest19656038191735390335.bam/part-r-00000.bam:0+133545
17:38:02.677 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:02.678 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:02.684 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:02.686 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:02.686 INFO Executor - Finished task 1.0 in stage 93.0 (TID 139). 989 bytes result sent to driver
17:38:02.688 INFO TaskSetManager - Finished task 1.0 in stage 93.0 (TID 139) in 62 ms on localhost (executor driver) (1/2)
17:38:02.690 INFO Executor - Finished task 0.0 in stage 93.0 (TID 138). 989 bytes result sent to driver
17:38:02.690 INFO TaskSetManager - Finished task 0.0 in stage 93.0 (TID 138) in 64 ms on localhost (executor driver) (2/2)
17:38:02.690 INFO TaskSchedulerImpl - Removed TaskSet 93.0, whose tasks have all completed, from pool
17:38:02.690 INFO DAGScheduler - ResultStage 93 (count at ReadsSparkSinkUnitTest.java:222) finished in 0.082 s
17:38:02.690 INFO DAGScheduler - Job 67 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:02.690 INFO TaskSchedulerImpl - Killing all running tasks in stage 93: Stage finished
17:38:02.690 INFO DAGScheduler - Job 67 finished: count at ReadsSparkSinkUnitTest.java:222, took 0.083599 s
17:38:02.692 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:02.694 INFO MemoryStore - Block broadcast_168 stored as values in memory (estimated size 297.9 KiB, free 1917.9 MiB)
17:38:02.699 INFO MemoryStore - Block broadcast_168_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1917.9 MiB)
17:38:02.700 INFO BlockManagerInfo - Added broadcast_168_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.6 MiB)
17:38:02.700 INFO SparkContext - Created broadcast 168 from newAPIHadoopFile at PathSplitSource.java:96
17:38:02.721 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:02.723 INFO MemoryStore - Block broadcast_169 stored as values in memory (estimated size 297.9 KiB, free 1917.6 MiB)
17:38:02.729 INFO MemoryStore - Block broadcast_169_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1917.5 MiB)
17:38:02.729 INFO BlockManagerInfo - Added broadcast_169_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.6 MiB)
17:38:02.729 INFO SparkContext - Created broadcast 169 from newAPIHadoopFile at PathSplitSource.java:96
17:38:02.748 INFO MemoryStore - Block broadcast_170 stored as values in memory (estimated size 160.7 KiB, free 1917.4 MiB)
17:38:02.749 INFO MemoryStore - Block broadcast_170_piece0 stored as bytes in memory (estimated size 9.6 KiB, free 1917.4 MiB)
17:38:02.749 INFO BlockManagerInfo - Added broadcast_170_piece0 in memory on localhost:37953 (size: 9.6 KiB, free: 1919.6 MiB)
17:38:02.749 INFO SparkContext - Created broadcast 170 from broadcast at ReadsSparkSink.java:133
17:38:02.751 INFO MemoryStore - Block broadcast_171 stored as values in memory (estimated size 163.2 KiB, free 1917.2 MiB)
17:38:02.752 INFO MemoryStore - Block broadcast_171_piece0 stored as bytes in memory (estimated size 9.6 KiB, free 1917.2 MiB)
17:38:02.752 INFO BlockManagerInfo - Added broadcast_171_piece0 in memory on localhost:37953 (size: 9.6 KiB, free: 1919.6 MiB)
17:38:02.752 INFO SparkContext - Created broadcast 171 from broadcast at AnySamSinkMultiple.java:80
17:38:02.754 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:02.754 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:02.754 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:02.763 INFO FileInputFormat - Total input files to process : 1
17:38:02.769 INFO SparkContext - Starting job: runJob at SparkHadoopWriter.scala:83
17:38:02.769 INFO DAGScheduler - Registering RDD 404 (repartition at ReadsSparkSinkUnitTest.java:210) as input to shuffle 20
17:38:02.769 INFO DAGScheduler - Got job 68 (runJob at SparkHadoopWriter.scala:83) with 2 output partitions
17:38:02.769 INFO DAGScheduler - Final stage: ResultStage 95 (runJob at SparkHadoopWriter.scala:83)
17:38:02.769 INFO DAGScheduler - Parents of final stage: List(ShuffleMapStage 94)
17:38:02.769 INFO DAGScheduler - Missing parents: List(ShuffleMapStage 94)
17:38:02.770 INFO DAGScheduler - Submitting ShuffleMapStage 94 (MapPartitionsRDD[404] at repartition at ReadsSparkSinkUnitTest.java:210), which has no missing parents
17:38:02.785 INFO MemoryStore - Block broadcast_172 stored as values in memory (estimated size 427.7 KiB, free 1916.8 MiB)
17:38:02.790 INFO BlockManagerInfo - Removed broadcast_166_piece0 on localhost:37953 in memory (size: 3.4 KiB, free: 1919.6 MiB)
17:38:02.790 INFO MemoryStore - Block broadcast_172_piece0 stored as bytes in memory (estimated size 154.6 KiB, free 1916.6 MiB)
17:38:02.791 INFO BlockManagerInfo - Added broadcast_172_piece0 in memory on localhost:37953 (size: 154.6 KiB, free: 1919.4 MiB)
17:38:02.791 INFO SparkContext - Created broadcast 172 from broadcast at DAGScheduler.scala:1580
17:38:02.791 INFO DAGScheduler - Submitting 1 missing tasks from ShuffleMapStage 94 (MapPartitionsRDD[404] at repartition at ReadsSparkSinkUnitTest.java:210) (first 15 tasks are for partitions Vector(0))
17:38:02.791 INFO TaskSchedulerImpl - Adding task set 94.0 with 1 tasks resource profile 0
17:38:02.791 INFO BlockManagerInfo - Removed broadcast_159_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.5 MiB)
17:38:02.792 INFO TaskSetManager - Starting task 0.0 in stage 94.0 (TID 140) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7861 bytes)
17:38:02.792 INFO Executor - Running task 0.0 in stage 94.0 (TID 140)
17:38:02.792 INFO BlockManagerInfo - Removed broadcast_165_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.5 MiB)
17:38:02.793 INFO BlockManagerInfo - Removed broadcast_169_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.6 MiB)
17:38:02.795 INFO BlockManagerInfo - Removed broadcast_167_piece0 on localhost:37953 in memory (size: 153.6 KiB, free: 1919.7 MiB)
17:38:02.796 INFO BlockManagerInfo - Removed broadcast_162_piece0 on localhost:37953 in memory (size: 9.6 KiB, free: 1919.7 MiB)
17:38:02.797 INFO BlockManagerInfo - Removed broadcast_164_piece0 on localhost:37953 in memory (size: 56.2 KiB, free: 1919.8 MiB)
17:38:02.797 INFO BlockManagerInfo - Removed broadcast_161_piece0 on localhost:37953 in memory (size: 9.6 KiB, free: 1919.8 MiB)
17:38:02.826 INFO NewHadoopRDD - Input split: file:/home/runner/work/gatk/gatk/src/test/resources/org/broadinstitute/hellbender/tools/BQSR/HiSeq.1mb.1RG.2k_lines.bam:0+222075
17:38:02.827 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:02.828 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:02.835 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:02.847 INFO Executor - Finished task 0.0 in stage 94.0 (TID 140). 1149 bytes result sent to driver
17:38:02.847 INFO TaskSetManager - Finished task 0.0 in stage 94.0 (TID 140) in 55 ms on localhost (executor driver) (1/1)
17:38:02.848 INFO TaskSchedulerImpl - Removed TaskSet 94.0, whose tasks have all completed, from pool
17:38:02.848 INFO DAGScheduler - ShuffleMapStage 94 (repartition at ReadsSparkSinkUnitTest.java:210) finished in 0.078 s
17:38:02.848 INFO DAGScheduler - looking for newly runnable stages
17:38:02.848 INFO DAGScheduler - running: HashSet()
17:38:02.848 INFO DAGScheduler - waiting: HashSet(ResultStage 95)
17:38:02.848 INFO DAGScheduler - failed: HashSet()
17:38:02.848 INFO DAGScheduler - Submitting ResultStage 95 (MapPartitionsRDD[416] at mapToPair at AnySamSinkMultiple.java:89), which has no missing parents
17:38:02.854 INFO MemoryStore - Block broadcast_173 stored as values in memory (estimated size 150.2 KiB, free 1918.6 MiB)
17:38:02.855 INFO MemoryStore - Block broadcast_173_piece0 stored as bytes in memory (estimated size 56.2 KiB, free 1918.6 MiB)
17:38:02.855 INFO BlockManagerInfo - Added broadcast_173_piece0 in memory on localhost:37953 (size: 56.2 KiB, free: 1919.7 MiB)
17:38:02.855 INFO SparkContext - Created broadcast 173 from broadcast at DAGScheduler.scala:1580
17:38:02.855 INFO DAGScheduler - Submitting 2 missing tasks from ResultStage 95 (MapPartitionsRDD[416] at mapToPair at AnySamSinkMultiple.java:89) (first 15 tasks are for partitions Vector(0, 1))
17:38:02.855 INFO TaskSchedulerImpl - Adding task set 95.0 with 2 tasks resource profile 0
17:38:02.856 INFO TaskSetManager - Starting task 0.0 in stage 95.0 (TID 141) (localhost, executor driver, partition 0, NODE_LOCAL, 7860 bytes)
17:38:02.856 INFO TaskSetManager - Starting task 1.0 in stage 95.0 (TID 142) (localhost, executor driver, partition 1, NODE_LOCAL, 7860 bytes)
17:38:02.857 INFO Executor - Running task 0.0 in stage 95.0 (TID 141)
17:38:02.857 INFO Executor - Running task 1.0 in stage 95.0 (TID 142)
17:38:02.862 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:02.862 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:02.862 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:02.862 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:02.862 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:02.862 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:02.862 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:02.862 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:02.862 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:02.862 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:02.862 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:02.863 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:02.869 WARN IntelDeflaterFactory - Intel Deflater not supported, using Java.util.zip.Deflater
17:38:02.870 INFO ShuffleBlockFetcherIterator - Getting 1 (176.4 KiB) non-empty blocks including 1 (176.4 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
17:38:02.870 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
17:38:02.871 WARN IntelDeflaterFactory - Intel Deflater not supported, using Java.util.zip.Deflater
17:38:02.872 INFO ShuffleBlockFetcherIterator - Getting 1 (176.4 KiB) non-empty blocks including 1 (176.4 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
17:38:02.872 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
17:38:02.879 INFO FileOutputCommitter - Saved output of task 'attempt_20251027173802426618987106514273_0416_r_000000_0' to file:/tmp/ReadsSparkSinkUnitTest114788486772865509741.bam/_temporary/0/task_20251027173802426618987106514273_0416_r_000000
17:38:02.879 INFO SparkHadoopMapRedUtil - attempt_20251027173802426618987106514273_0416_r_000000_0: Committed. Elapsed time: 0 ms.
17:38:02.880 INFO FileOutputCommitter - Saved output of task 'attempt_20251027173802426618987106514273_0416_r_000001_0' to file:/tmp/ReadsSparkSinkUnitTest114788486772865509741.bam/_temporary/0/task_20251027173802426618987106514273_0416_r_000001
17:38:02.880 INFO SparkHadoopMapRedUtil - attempt_20251027173802426618987106514273_0416_r_000001_0: Committed. Elapsed time: 0 ms.
17:38:02.880 INFO Executor - Finished task 0.0 in stage 95.0 (TID 141). 1729 bytes result sent to driver
17:38:02.880 INFO Executor - Finished task 1.0 in stage 95.0 (TID 142). 1729 bytes result sent to driver
17:38:02.880 INFO TaskSetManager - Finished task 0.0 in stage 95.0 (TID 141) in 24 ms on localhost (executor driver) (1/2)
17:38:02.881 INFO TaskSetManager - Finished task 1.0 in stage 95.0 (TID 142) in 25 ms on localhost (executor driver) (2/2)
17:38:02.881 INFO TaskSchedulerImpl - Removed TaskSet 95.0, whose tasks have all completed, from pool
17:38:02.881 INFO DAGScheduler - ResultStage 95 (runJob at SparkHadoopWriter.scala:83) finished in 0.033 s
17:38:02.881 INFO DAGScheduler - Job 68 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:02.881 INFO TaskSchedulerImpl - Killing all running tasks in stage 95: Stage finished
17:38:02.881 INFO DAGScheduler - Job 68 finished: runJob at SparkHadoopWriter.scala:83, took 0.112423 s
17:38:02.881 INFO SparkHadoopWriter - Start to commit write Job job_20251027173802426618987106514273_0416.
17:38:02.885 INFO SparkHadoopWriter - Write Job job_20251027173802426618987106514273_0416 committed. Elapsed time: 3 ms.
17:38:02.886 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:02.888 INFO MemoryStore - Block broadcast_174 stored as values in memory (estimated size 297.9 KiB, free 1918.3 MiB)
17:38:02.894 INFO MemoryStore - Block broadcast_174_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1918.2 MiB)
17:38:02.894 INFO BlockManagerInfo - Added broadcast_174_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.7 MiB)
17:38:02.894 INFO SparkContext - Created broadcast 174 from newAPIHadoopFile at PathSplitSource.java:96
17:38:02.916 INFO SparkContext - Starting job: count at ReadsSparkSinkUnitTest.java:222
17:38:02.917 INFO DAGScheduler - Got job 69 (count at ReadsSparkSinkUnitTest.java:222) with 2 output partitions
17:38:02.917 INFO DAGScheduler - Final stage: ResultStage 97 (count at ReadsSparkSinkUnitTest.java:222)
17:38:02.917 INFO DAGScheduler - Parents of final stage: List(ShuffleMapStage 96)
17:38:02.917 INFO DAGScheduler - Missing parents: List()
17:38:02.917 INFO DAGScheduler - Submitting ResultStage 97 (MapPartitionsRDD[407] at repartition at ReadsSparkSinkUnitTest.java:210), which has no missing parents
17:38:02.918 INFO MemoryStore - Block broadcast_175 stored as values in memory (estimated size 6.3 KiB, free 1918.2 MiB)
17:38:02.918 INFO MemoryStore - Block broadcast_175_piece0 stored as bytes in memory (estimated size 3.4 KiB, free 1918.2 MiB)
17:38:02.918 INFO BlockManagerInfo - Added broadcast_175_piece0 in memory on localhost:37953 (size: 3.4 KiB, free: 1919.7 MiB)
17:38:02.919 INFO SparkContext - Created broadcast 175 from broadcast at DAGScheduler.scala:1580
17:38:02.919 INFO DAGScheduler - Submitting 2 missing tasks from ResultStage 97 (MapPartitionsRDD[407] at repartition at ReadsSparkSinkUnitTest.java:210) (first 15 tasks are for partitions Vector(0, 1))
17:38:02.919 INFO TaskSchedulerImpl - Adding task set 97.0 with 2 tasks resource profile 0
17:38:02.920 INFO TaskSetManager - Starting task 0.0 in stage 97.0 (TID 143) (localhost, executor driver, partition 0, NODE_LOCAL, 7860 bytes)
17:38:02.920 INFO TaskSetManager - Starting task 1.0 in stage 97.0 (TID 144) (localhost, executor driver, partition 1, NODE_LOCAL, 7860 bytes)
17:38:02.920 INFO Executor - Running task 0.0 in stage 97.0 (TID 143)
17:38:02.920 INFO Executor - Running task 1.0 in stage 97.0 (TID 144)
17:38:02.922 INFO ShuffleBlockFetcherIterator - Getting 1 (176.4 KiB) non-empty blocks including 1 (176.4 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
17:38:02.922 INFO ShuffleBlockFetcherIterator - Getting 1 (176.4 KiB) non-empty blocks including 1 (176.4 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
17:38:02.922 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
17:38:02.922 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
17:38:02.925 INFO Executor - Finished task 1.0 in stage 97.0 (TID 144). 1591 bytes result sent to driver
17:38:02.925 INFO Executor - Finished task 0.0 in stage 97.0 (TID 143). 1591 bytes result sent to driver
17:38:02.926 INFO TaskSetManager - Finished task 0.0 in stage 97.0 (TID 143) in 6 ms on localhost (executor driver) (1/2)
17:38:02.926 INFO TaskSetManager - Finished task 1.0 in stage 97.0 (TID 144) in 6 ms on localhost (executor driver) (2/2)
17:38:02.926 INFO TaskSchedulerImpl - Removed TaskSet 97.0, whose tasks have all completed, from pool
17:38:02.926 INFO DAGScheduler - ResultStage 97 (count at ReadsSparkSinkUnitTest.java:222) finished in 0.008 s
17:38:02.926 INFO DAGScheduler - Job 69 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:02.926 INFO TaskSchedulerImpl - Killing all running tasks in stage 97: Stage finished
17:38:02.926 INFO DAGScheduler - Job 69 finished: count at ReadsSparkSinkUnitTest.java:222, took 0.009855 s
17:38:02.935 INFO FileInputFormat - Total input files to process : 2
17:38:02.938 INFO SparkContext - Starting job: count at ReadsSparkSinkUnitTest.java:222
17:38:02.938 INFO DAGScheduler - Got job 70 (count at ReadsSparkSinkUnitTest.java:222) with 2 output partitions
17:38:02.938 INFO DAGScheduler - Final stage: ResultStage 98 (count at ReadsSparkSinkUnitTest.java:222)
17:38:02.938 INFO DAGScheduler - Parents of final stage: List()
17:38:02.938 INFO DAGScheduler - Missing parents: List()
17:38:02.938 INFO DAGScheduler - Submitting ResultStage 98 (MapPartitionsRDD[423] at filter at ReadsSparkSource.java:96), which has no missing parents
17:38:02.954 INFO MemoryStore - Block broadcast_176 stored as values in memory (estimated size 426.1 KiB, free 1917.8 MiB)
17:38:02.955 INFO MemoryStore - Block broadcast_176_piece0 stored as bytes in memory (estimated size 153.6 KiB, free 1917.6 MiB)
17:38:02.955 INFO BlockManagerInfo - Added broadcast_176_piece0 in memory on localhost:37953 (size: 153.6 KiB, free: 1919.5 MiB)
17:38:02.956 INFO SparkContext - Created broadcast 176 from broadcast at DAGScheduler.scala:1580
17:38:02.956 INFO DAGScheduler - Submitting 2 missing tasks from ResultStage 98 (MapPartitionsRDD[423] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0, 1))
17:38:02.956 INFO TaskSchedulerImpl - Adding task set 98.0 with 2 tasks resource profile 0
17:38:02.957 INFO TaskSetManager - Starting task 0.0 in stage 98.0 (TID 145) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7827 bytes)
17:38:02.957 INFO TaskSetManager - Starting task 1.0 in stage 98.0 (TID 146) (localhost, executor driver, partition 1, PROCESS_LOCAL, 7827 bytes)
17:38:02.957 INFO Executor - Running task 0.0 in stage 98.0 (TID 145)
17:38:02.957 INFO Executor - Running task 1.0 in stage 98.0 (TID 146)
17:38:02.992 INFO NewHadoopRDD - Input split: file:/tmp/ReadsSparkSinkUnitTest114788486772865509741.bam/part-r-00001.bam:0+129979
17:38:02.992 INFO NewHadoopRDD - Input split: file:/tmp/ReadsSparkSinkUnitTest114788486772865509741.bam/part-r-00000.bam:0+133545
17:38:02.993 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:02.993 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:02.994 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:02.994 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:03.001 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:03.001 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:03.004 INFO Executor - Finished task 0.0 in stage 98.0 (TID 145). 989 bytes result sent to driver
17:38:03.004 INFO Executor - Finished task 1.0 in stage 98.0 (TID 146). 989 bytes result sent to driver
17:38:03.004 INFO TaskSetManager - Finished task 0.0 in stage 98.0 (TID 145) in 48 ms on localhost (executor driver) (1/2)
17:38:03.004 INFO TaskSetManager - Finished task 1.0 in stage 98.0 (TID 146) in 47 ms on localhost (executor driver) (2/2)
17:38:03.005 INFO TaskSchedulerImpl - Removed TaskSet 98.0, whose tasks have all completed, from pool
17:38:03.005 INFO DAGScheduler - ResultStage 98 (count at ReadsSparkSinkUnitTest.java:222) finished in 0.066 s
17:38:03.005 INFO DAGScheduler - Job 70 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:03.005 INFO TaskSchedulerImpl - Killing all running tasks in stage 98: Stage finished
17:38:03.005 INFO DAGScheduler - Job 70 finished: count at ReadsSparkSinkUnitTest.java:222, took 0.066878 s
17:38:03.006 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:03.008 INFO MemoryStore - Block broadcast_177 stored as values in memory (estimated size 298.0 KiB, free 1917.3 MiB)
17:38:03.014 INFO MemoryStore - Block broadcast_177_piece0 stored as bytes in memory (estimated size 50.3 KiB, free 1917.3 MiB)
17:38:03.014 INFO BlockManagerInfo - Added broadcast_177_piece0 in memory on localhost:37953 (size: 50.3 KiB, free: 1919.5 MiB)
17:38:03.014 INFO SparkContext - Created broadcast 177 from newAPIHadoopFile at PathSplitSource.java:96
17:38:03.035 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:03.037 INFO MemoryStore - Block broadcast_178 stored as values in memory (estimated size 298.0 KiB, free 1917.0 MiB)
17:38:03.045 INFO BlockManagerInfo - Removed broadcast_168_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.5 MiB)
17:38:03.047 INFO BlockManagerInfo - Removed broadcast_172_piece0 on localhost:37953 in memory (size: 154.6 KiB, free: 1919.7 MiB)
17:38:03.047 INFO MemoryStore - Block broadcast_178_piece0 stored as bytes in memory (estimated size 50.3 KiB, free 1917.9 MiB)
17:38:03.047 INFO BlockManagerInfo - Added broadcast_178_piece0 in memory on localhost:37953 (size: 50.3 KiB, free: 1919.6 MiB)
17:38:03.048 INFO BlockManagerInfo - Removed broadcast_176_piece0 on localhost:37953 in memory (size: 153.6 KiB, free: 1919.8 MiB)
17:38:03.048 INFO SparkContext - Created broadcast 178 from newAPIHadoopFile at PathSplitSource.java:96
17:38:03.049 INFO BlockManagerInfo - Removed broadcast_171_piece0 on localhost:37953 in memory (size: 9.6 KiB, free: 1919.8 MiB)
17:38:03.051 INFO BlockManagerInfo - Removed broadcast_170_piece0 on localhost:37953 in memory (size: 9.6 KiB, free: 1919.8 MiB)
17:38:03.051 INFO BlockManagerInfo - Removed broadcast_175_piece0 on localhost:37953 in memory (size: 3.4 KiB, free: 1919.8 MiB)
17:38:03.052 INFO BlockManagerInfo - Removed broadcast_173_piece0 on localhost:37953 in memory (size: 56.2 KiB, free: 1919.9 MiB)
17:38:03.053 INFO BlockManagerInfo - Removed broadcast_174_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.9 MiB)
17:38:03.067 INFO MemoryStore - Block broadcast_179 stored as values in memory (estimated size 160.7 KiB, free 1919.2 MiB)
17:38:03.068 INFO MemoryStore - Block broadcast_179_piece0 stored as bytes in memory (estimated size 9.6 KiB, free 1919.2 MiB)
17:38:03.068 INFO BlockManagerInfo - Added broadcast_179_piece0 in memory on localhost:37953 (size: 9.6 KiB, free: 1919.9 MiB)
17:38:03.068 INFO SparkContext - Created broadcast 179 from broadcast at ReadsSparkSink.java:133
17:38:03.070 INFO MemoryStore - Block broadcast_180 stored as values in memory (estimated size 163.2 KiB, free 1919.0 MiB)
17:38:03.070 INFO MemoryStore - Block broadcast_180_piece0 stored as bytes in memory (estimated size 9.6 KiB, free 1919.0 MiB)
17:38:03.070 INFO BlockManagerInfo - Added broadcast_180_piece0 in memory on localhost:37953 (size: 9.6 KiB, free: 1919.9 MiB)
17:38:03.071 INFO SparkContext - Created broadcast 180 from broadcast at AnySamSinkMultiple.java:80
17:38:03.073 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:03.073 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:03.073 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:03.081 INFO FileInputFormat - Total input files to process : 1
17:38:03.087 INFO SparkContext - Starting job: runJob at SparkHadoopWriter.scala:83
17:38:03.088 INFO DAGScheduler - Registering RDD 431 (repartition at ReadsSparkSinkUnitTest.java:210) as input to shuffle 21
17:38:03.088 INFO DAGScheduler - Got job 71 (runJob at SparkHadoopWriter.scala:83) with 2 output partitions
17:38:03.088 INFO DAGScheduler - Final stage: ResultStage 100 (runJob at SparkHadoopWriter.scala:83)
17:38:03.088 INFO DAGScheduler - Parents of final stage: List(ShuffleMapStage 99)
17:38:03.088 INFO DAGScheduler - Missing parents: List(ShuffleMapStage 99)
17:38:03.088 INFO DAGScheduler - Submitting ShuffleMapStage 99 (MapPartitionsRDD[431] at repartition at ReadsSparkSinkUnitTest.java:210), which has no missing parents
17:38:03.104 INFO MemoryStore - Block broadcast_181 stored as values in memory (estimated size 427.7 KiB, free 1918.6 MiB)
17:38:03.105 INFO MemoryStore - Block broadcast_181_piece0 stored as bytes in memory (estimated size 154.6 KiB, free 1918.4 MiB)
17:38:03.105 INFO BlockManagerInfo - Added broadcast_181_piece0 in memory on localhost:37953 (size: 154.6 KiB, free: 1919.7 MiB)
17:38:03.105 INFO SparkContext - Created broadcast 181 from broadcast at DAGScheduler.scala:1580
17:38:03.105 INFO DAGScheduler - Submitting 1 missing tasks from ShuffleMapStage 99 (MapPartitionsRDD[431] at repartition at ReadsSparkSinkUnitTest.java:210) (first 15 tasks are for partitions Vector(0))
17:38:03.105 INFO TaskSchedulerImpl - Adding task set 99.0 with 1 tasks resource profile 0
17:38:03.106 INFO TaskSetManager - Starting task 0.0 in stage 99.0 (TID 147) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7901 bytes)
17:38:03.106 INFO Executor - Running task 0.0 in stage 99.0 (TID 147)
17:38:03.138 INFO NewHadoopRDD - Input split: file:/home/runner/work/gatk/gatk/src/test/resources/org/broadinstitute/hellbender/tools/BQSR/expected.HiSeq.1mb.1RG.2k_lines.alternate.recalibrated.DIQ.bam:0+216896
17:38:03.139 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:03.140 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:03.149 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:03.160 INFO Executor - Finished task 0.0 in stage 99.0 (TID 147). 1149 bytes result sent to driver
17:38:03.161 INFO TaskSetManager - Finished task 0.0 in stage 99.0 (TID 147) in 55 ms on localhost (executor driver) (1/1)
17:38:03.161 INFO TaskSchedulerImpl - Removed TaskSet 99.0, whose tasks have all completed, from pool
17:38:03.161 INFO DAGScheduler - ShuffleMapStage 99 (repartition at ReadsSparkSinkUnitTest.java:210) finished in 0.073 s
17:38:03.161 INFO DAGScheduler - looking for newly runnable stages
17:38:03.161 INFO DAGScheduler - running: HashSet()
17:38:03.161 INFO DAGScheduler - waiting: HashSet(ResultStage 100)
17:38:03.161 INFO DAGScheduler - failed: HashSet()
17:38:03.161 INFO DAGScheduler - Submitting ResultStage 100 (MapPartitionsRDD[443] at mapToPair at AnySamSinkMultiple.java:89), which has no missing parents
17:38:03.167 INFO MemoryStore - Block broadcast_182 stored as values in memory (estimated size 150.2 KiB, free 1918.3 MiB)
17:38:03.168 INFO MemoryStore - Block broadcast_182_piece0 stored as bytes in memory (estimated size 56.2 KiB, free 1918.2 MiB)
17:38:03.168 INFO BlockManagerInfo - Added broadcast_182_piece0 in memory on localhost:37953 (size: 56.2 KiB, free: 1919.7 MiB)
17:38:03.168 INFO SparkContext - Created broadcast 182 from broadcast at DAGScheduler.scala:1580
17:38:03.168 INFO DAGScheduler - Submitting 2 missing tasks from ResultStage 100 (MapPartitionsRDD[443] at mapToPair at AnySamSinkMultiple.java:89) (first 15 tasks are for partitions Vector(0, 1))
17:38:03.168 INFO TaskSchedulerImpl - Adding task set 100.0 with 2 tasks resource profile 0
17:38:03.169 INFO TaskSetManager - Starting task 0.0 in stage 100.0 (TID 148) (localhost, executor driver, partition 0, NODE_LOCAL, 7860 bytes)
17:38:03.169 INFO TaskSetManager - Starting task 1.0 in stage 100.0 (TID 149) (localhost, executor driver, partition 1, NODE_LOCAL, 7860 bytes)
17:38:03.170 INFO Executor - Running task 0.0 in stage 100.0 (TID 148)
17:38:03.170 INFO Executor - Running task 1.0 in stage 100.0 (TID 149)
17:38:03.175 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:03.175 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:03.175 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:03.175 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:03.175 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:03.175 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:03.175 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:03.175 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:03.175 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:03.175 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:03.175 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:03.175 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:03.181 WARN IntelDeflaterFactory - Intel Deflater not supported, using Java.util.zip.Deflater
17:38:03.182 INFO ShuffleBlockFetcherIterator - Getting 1 (176.4 KiB) non-empty blocks including 1 (176.4 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
17:38:03.182 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
17:38:03.183 WARN IntelDeflaterFactory - Intel Deflater not supported, using Java.util.zip.Deflater
17:38:03.184 INFO ShuffleBlockFetcherIterator - Getting 1 (176.4 KiB) non-empty blocks including 1 (176.4 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
17:38:03.184 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
17:38:03.190 INFO FileOutputCommitter - Saved output of task 'attempt_202510271738032276809848822791227_0443_r_000001_0' to file:/tmp/ReadsSparkSinkUnitTest22799701088740774412.bam/_temporary/0/task_202510271738032276809848822791227_0443_r_000001
17:38:03.190 INFO SparkHadoopMapRedUtil - attempt_202510271738032276809848822791227_0443_r_000001_0: Committed. Elapsed time: 0 ms.
17:38:03.190 INFO Executor - Finished task 1.0 in stage 100.0 (TID 149). 1729 bytes result sent to driver
17:38:03.191 INFO TaskSetManager - Finished task 1.0 in stage 100.0 (TID 149) in 22 ms on localhost (executor driver) (1/2)
17:38:03.191 INFO FileOutputCommitter - Saved output of task 'attempt_202510271738032276809848822791227_0443_r_000000_0' to file:/tmp/ReadsSparkSinkUnitTest22799701088740774412.bam/_temporary/0/task_202510271738032276809848822791227_0443_r_000000
17:38:03.191 INFO SparkHadoopMapRedUtil - attempt_202510271738032276809848822791227_0443_r_000000_0: Committed. Elapsed time: 0 ms.
17:38:03.192 INFO Executor - Finished task 0.0 in stage 100.0 (TID 148). 1729 bytes result sent to driver
17:38:03.192 INFO TaskSetManager - Finished task 0.0 in stage 100.0 (TID 148) in 23 ms on localhost (executor driver) (2/2)
17:38:03.192 INFO TaskSchedulerImpl - Removed TaskSet 100.0, whose tasks have all completed, from pool
17:38:03.192 INFO DAGScheduler - ResultStage 100 (runJob at SparkHadoopWriter.scala:83) finished in 0.030 s
17:38:03.192 INFO DAGScheduler - Job 71 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:03.192 INFO TaskSchedulerImpl - Killing all running tasks in stage 100: Stage finished
17:38:03.192 INFO DAGScheduler - Job 71 finished: runJob at SparkHadoopWriter.scala:83, took 0.105196 s
17:38:03.193 INFO SparkHadoopWriter - Start to commit write Job job_202510271738032276809848822791227_0443.
17:38:03.197 INFO SparkHadoopWriter - Write Job job_202510271738032276809848822791227_0443 committed. Elapsed time: 3 ms.
17:38:03.197 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:03.199 INFO MemoryStore - Block broadcast_183 stored as values in memory (estimated size 297.9 KiB, free 1917.9 MiB)
17:38:03.205 INFO MemoryStore - Block broadcast_183_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1917.9 MiB)
17:38:03.205 INFO BlockManagerInfo - Added broadcast_183_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.6 MiB)
17:38:03.205 INFO SparkContext - Created broadcast 183 from newAPIHadoopFile at PathSplitSource.java:96
17:38:03.227 INFO SparkContext - Starting job: count at ReadsSparkSinkUnitTest.java:222
17:38:03.228 INFO DAGScheduler - Got job 72 (count at ReadsSparkSinkUnitTest.java:222) with 2 output partitions
17:38:03.228 INFO DAGScheduler - Final stage: ResultStage 102 (count at ReadsSparkSinkUnitTest.java:222)
17:38:03.228 INFO DAGScheduler - Parents of final stage: List(ShuffleMapStage 101)
17:38:03.228 INFO DAGScheduler - Missing parents: List()
17:38:03.228 INFO DAGScheduler - Submitting ResultStage 102 (MapPartitionsRDD[434] at repartition at ReadsSparkSinkUnitTest.java:210), which has no missing parents
17:38:03.229 INFO MemoryStore - Block broadcast_184 stored as values in memory (estimated size 6.3 KiB, free 1917.9 MiB)
17:38:03.229 INFO MemoryStore - Block broadcast_184_piece0 stored as bytes in memory (estimated size 3.4 KiB, free 1917.9 MiB)
17:38:03.229 INFO BlockManagerInfo - Added broadcast_184_piece0 in memory on localhost:37953 (size: 3.4 KiB, free: 1919.6 MiB)
17:38:03.229 INFO SparkContext - Created broadcast 184 from broadcast at DAGScheduler.scala:1580
17:38:03.230 INFO DAGScheduler - Submitting 2 missing tasks from ResultStage 102 (MapPartitionsRDD[434] at repartition at ReadsSparkSinkUnitTest.java:210) (first 15 tasks are for partitions Vector(0, 1))
17:38:03.230 INFO TaskSchedulerImpl - Adding task set 102.0 with 2 tasks resource profile 0
17:38:03.230 INFO TaskSetManager - Starting task 0.0 in stage 102.0 (TID 150) (localhost, executor driver, partition 0, NODE_LOCAL, 7860 bytes)
17:38:03.230 INFO TaskSetManager - Starting task 1.0 in stage 102.0 (TID 151) (localhost, executor driver, partition 1, NODE_LOCAL, 7860 bytes)
17:38:03.231 INFO Executor - Running task 0.0 in stage 102.0 (TID 150)
17:38:03.231 INFO Executor - Running task 1.0 in stage 102.0 (TID 151)
17:38:03.232 INFO ShuffleBlockFetcherIterator - Getting 1 (176.4 KiB) non-empty blocks including 1 (176.4 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
17:38:03.232 INFO ShuffleBlockFetcherIterator - Getting 1 (176.4 KiB) non-empty blocks including 1 (176.4 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
17:38:03.232 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
17:38:03.232 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
17:38:03.236 INFO Executor - Finished task 1.0 in stage 102.0 (TID 151). 1591 bytes result sent to driver
17:38:03.236 INFO Executor - Finished task 0.0 in stage 102.0 (TID 150). 1591 bytes result sent to driver
17:38:03.236 INFO TaskSetManager - Finished task 0.0 in stage 102.0 (TID 150) in 6 ms on localhost (executor driver) (1/2)
17:38:03.236 INFO TaskSetManager - Finished task 1.0 in stage 102.0 (TID 151) in 6 ms on localhost (executor driver) (2/2)
17:38:03.236 INFO TaskSchedulerImpl - Removed TaskSet 102.0, whose tasks have all completed, from pool
17:38:03.236 INFO DAGScheduler - ResultStage 102 (count at ReadsSparkSinkUnitTest.java:222) finished in 0.008 s
17:38:03.237 INFO DAGScheduler - Job 72 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:03.237 INFO TaskSchedulerImpl - Killing all running tasks in stage 102: Stage finished
17:38:03.237 INFO DAGScheduler - Job 72 finished: count at ReadsSparkSinkUnitTest.java:222, took 0.009407 s
17:38:03.245 INFO FileInputFormat - Total input files to process : 2
17:38:03.248 INFO SparkContext - Starting job: count at ReadsSparkSinkUnitTest.java:222
17:38:03.248 INFO DAGScheduler - Got job 73 (count at ReadsSparkSinkUnitTest.java:222) with 2 output partitions
17:38:03.248 INFO DAGScheduler - Final stage: ResultStage 103 (count at ReadsSparkSinkUnitTest.java:222)
17:38:03.248 INFO DAGScheduler - Parents of final stage: List()
17:38:03.248 INFO DAGScheduler - Missing parents: List()
17:38:03.249 INFO DAGScheduler - Submitting ResultStage 103 (MapPartitionsRDD[450] at filter at ReadsSparkSource.java:96), which has no missing parents
17:38:03.264 INFO MemoryStore - Block broadcast_185 stored as values in memory (estimated size 426.1 KiB, free 1917.4 MiB)
17:38:03.269 INFO BlockManagerInfo - Removed broadcast_180_piece0 on localhost:37953 in memory (size: 9.6 KiB, free: 1919.6 MiB)
17:38:03.269 INFO MemoryStore - Block broadcast_185_piece0 stored as bytes in memory (estimated size 153.6 KiB, free 1917.5 MiB)
17:38:03.270 INFO BlockManagerInfo - Added broadcast_185_piece0 in memory on localhost:37953 (size: 153.6 KiB, free: 1919.5 MiB)
17:38:03.270 INFO BlockManagerInfo - Removed broadcast_181_piece0 on localhost:37953 in memory (size: 154.6 KiB, free: 1919.6 MiB)
17:38:03.271 INFO BlockManagerInfo - Removed broadcast_182_piece0 on localhost:37953 in memory (size: 56.2 KiB, free: 1919.7 MiB)
17:38:03.272 INFO SparkContext - Created broadcast 185 from broadcast at DAGScheduler.scala:1580
17:38:03.272 INFO DAGScheduler - Submitting 2 missing tasks from ResultStage 103 (MapPartitionsRDD[450] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0, 1))
17:38:03.272 INFO TaskSchedulerImpl - Adding task set 103.0 with 2 tasks resource profile 0
17:38:03.273 INFO TaskSetManager - Starting task 0.0 in stage 103.0 (TID 152) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7826 bytes)
17:38:03.273 INFO TaskSetManager - Starting task 1.0 in stage 103.0 (TID 153) (localhost, executor driver, partition 1, PROCESS_LOCAL, 7826 bytes)
17:38:03.273 INFO BlockManagerInfo - Removed broadcast_178_piece0 on localhost:37953 in memory (size: 50.3 KiB, free: 1919.7 MiB)
17:38:03.273 INFO Executor - Running task 1.0 in stage 103.0 (TID 153)
17:38:03.273 INFO Executor - Running task 0.0 in stage 103.0 (TID 152)
17:38:03.276 INFO BlockManagerInfo - Removed broadcast_184_piece0 on localhost:37953 in memory (size: 3.4 KiB, free: 1919.7 MiB)
17:38:03.277 INFO BlockManagerInfo - Removed broadcast_179_piece0 on localhost:37953 in memory (size: 9.6 KiB, free: 1919.8 MiB)
17:38:03.311 INFO NewHadoopRDD - Input split: file:/tmp/ReadsSparkSinkUnitTest22799701088740774412.bam/part-r-00000.bam:0+129752
17:38:03.311 INFO NewHadoopRDD - Input split: file:/tmp/ReadsSparkSinkUnitTest22799701088740774412.bam/part-r-00001.bam:0+129515
17:38:03.312 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:03.312 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:03.313 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:03.313 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:03.321 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:03.321 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:03.323 INFO Executor - Finished task 0.0 in stage 103.0 (TID 152). 989 bytes result sent to driver
17:38:03.323 INFO Executor - Finished task 1.0 in stage 103.0 (TID 153). 989 bytes result sent to driver
17:38:03.324 INFO TaskSetManager - Finished task 1.0 in stage 103.0 (TID 153) in 51 ms on localhost (executor driver) (1/2)
17:38:03.324 INFO TaskSetManager - Finished task 0.0 in stage 103.0 (TID 152) in 51 ms on localhost (executor driver) (2/2)
17:38:03.324 INFO TaskSchedulerImpl - Removed TaskSet 103.0, whose tasks have all completed, from pool
17:38:03.324 INFO DAGScheduler - ResultStage 103 (count at ReadsSparkSinkUnitTest.java:222) finished in 0.075 s
17:38:03.325 INFO DAGScheduler - Job 73 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:03.325 INFO TaskSchedulerImpl - Killing all running tasks in stage 103: Stage finished
17:38:03.325 INFO DAGScheduler - Job 73 finished: count at ReadsSparkSinkUnitTest.java:222, took 0.076652 s
17:38:03.326 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:03.328 INFO MemoryStore - Block broadcast_186 stored as values in memory (estimated size 298.0 KiB, free 1918.5 MiB)
17:38:03.334 INFO MemoryStore - Block broadcast_186_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1918.4 MiB)
17:38:03.334 INFO BlockManagerInfo - Added broadcast_186_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.7 MiB)
17:38:03.334 INFO SparkContext - Created broadcast 186 from newAPIHadoopFile at PathSplitSource.java:96
17:38:03.355 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:03.357 INFO MemoryStore - Block broadcast_187 stored as values in memory (estimated size 298.0 KiB, free 1918.1 MiB)
17:38:03.363 INFO MemoryStore - Block broadcast_187_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1918.1 MiB)
17:38:03.363 INFO BlockManagerInfo - Added broadcast_187_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.7 MiB)
17:38:03.363 INFO SparkContext - Created broadcast 187 from newAPIHadoopFile at PathSplitSource.java:96
17:38:03.382 INFO MemoryStore - Block broadcast_188 stored as values in memory (estimated size 19.6 KiB, free 1918.1 MiB)
17:38:03.383 INFO MemoryStore - Block broadcast_188_piece0 stored as bytes in memory (estimated size 1890.0 B, free 1918.1 MiB)
17:38:03.383 INFO BlockManagerInfo - Added broadcast_188_piece0 in memory on localhost:37953 (size: 1890.0 B, free: 1919.7 MiB)
17:38:03.383 INFO SparkContext - Created broadcast 188 from broadcast at ReadsSparkSink.java:133
17:38:03.384 INFO MemoryStore - Block broadcast_189 stored as values in memory (estimated size 20.0 KiB, free 1918.0 MiB)
17:38:03.385 INFO MemoryStore - Block broadcast_189_piece0 stored as bytes in memory (estimated size 1890.0 B, free 1918.0 MiB)
17:38:03.385 INFO BlockManagerInfo - Added broadcast_189_piece0 in memory on localhost:37953 (size: 1890.0 B, free: 1919.7 MiB)
17:38:03.385 INFO SparkContext - Created broadcast 189 from broadcast at AnySamSinkMultiple.java:80
17:38:03.387 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:03.387 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:03.387 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:03.396 INFO FileInputFormat - Total input files to process : 1
17:38:03.402 INFO SparkContext - Starting job: runJob at SparkHadoopWriter.scala:83
17:38:03.403 INFO DAGScheduler - Registering RDD 458 (repartition at ReadsSparkSinkUnitTest.java:210) as input to shuffle 22
17:38:03.403 INFO DAGScheduler - Got job 74 (runJob at SparkHadoopWriter.scala:83) with 2 output partitions
17:38:03.403 INFO DAGScheduler - Final stage: ResultStage 105 (runJob at SparkHadoopWriter.scala:83)
17:38:03.403 INFO DAGScheduler - Parents of final stage: List(ShuffleMapStage 104)
17:38:03.403 INFO DAGScheduler - Missing parents: List(ShuffleMapStage 104)
17:38:03.403 INFO DAGScheduler - Submitting ShuffleMapStage 104 (MapPartitionsRDD[458] at repartition at ReadsSparkSinkUnitTest.java:210), which has no missing parents
17:38:03.419 INFO MemoryStore - Block broadcast_190 stored as values in memory (estimated size 427.7 KiB, free 1917.6 MiB)
17:38:03.421 INFO MemoryStore - Block broadcast_190_piece0 stored as bytes in memory (estimated size 154.6 KiB, free 1917.5 MiB)
17:38:03.421 INFO BlockManagerInfo - Added broadcast_190_piece0 in memory on localhost:37953 (size: 154.6 KiB, free: 1919.5 MiB)
17:38:03.421 INFO SparkContext - Created broadcast 190 from broadcast at DAGScheduler.scala:1580
17:38:03.421 INFO DAGScheduler - Submitting 1 missing tasks from ShuffleMapStage 104 (MapPartitionsRDD[458] at repartition at ReadsSparkSinkUnitTest.java:210) (first 15 tasks are for partitions Vector(0))
17:38:03.421 INFO TaskSchedulerImpl - Adding task set 104.0 with 1 tasks resource profile 0
17:38:03.422 INFO TaskSetManager - Starting task 0.0 in stage 104.0 (TID 154) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7882 bytes)
17:38:03.422 INFO Executor - Running task 0.0 in stage 104.0 (TID 154)
17:38:03.456 INFO NewHadoopRDD - Input split: file:/home/runner/work/gatk/gatk/src/test/resources/org/broadinstitute/hellbender/tools/BQSR/CEUTrio.HiSeq.WGS.b37.ch20.1m-1m1k.NA12878.bam:0+211123
17:38:03.457 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:03.457 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:03.463 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:03.473 INFO Executor - Finished task 0.0 in stage 104.0 (TID 154). 1149 bytes result sent to driver
17:38:03.474 INFO TaskSetManager - Finished task 0.0 in stage 104.0 (TID 154) in 52 ms on localhost (executor driver) (1/1)
17:38:03.474 INFO TaskSchedulerImpl - Removed TaskSet 104.0, whose tasks have all completed, from pool
17:38:03.474 INFO DAGScheduler - ShuffleMapStage 104 (repartition at ReadsSparkSinkUnitTest.java:210) finished in 0.070 s
17:38:03.474 INFO DAGScheduler - looking for newly runnable stages
17:38:03.474 INFO DAGScheduler - running: HashSet()
17:38:03.474 INFO DAGScheduler - waiting: HashSet(ResultStage 105)
17:38:03.474 INFO DAGScheduler - failed: HashSet()
17:38:03.474 INFO DAGScheduler - Submitting ResultStage 105 (MapPartitionsRDD[470] at mapToPair at AnySamSinkMultiple.java:89), which has no missing parents
17:38:03.481 INFO MemoryStore - Block broadcast_191 stored as values in memory (estimated size 150.2 KiB, free 1917.3 MiB)
17:38:03.481 INFO MemoryStore - Block broadcast_191_piece0 stored as bytes in memory (estimated size 56.2 KiB, free 1917.3 MiB)
17:38:03.481 INFO BlockManagerInfo - Added broadcast_191_piece0 in memory on localhost:37953 (size: 56.2 KiB, free: 1919.4 MiB)
17:38:03.482 INFO SparkContext - Created broadcast 191 from broadcast at DAGScheduler.scala:1580
17:38:03.482 INFO DAGScheduler - Submitting 2 missing tasks from ResultStage 105 (MapPartitionsRDD[470] at mapToPair at AnySamSinkMultiple.java:89) (first 15 tasks are for partitions Vector(0, 1))
17:38:03.482 INFO TaskSchedulerImpl - Adding task set 105.0 with 2 tasks resource profile 0
17:38:03.483 INFO TaskSetManager - Starting task 0.0 in stage 105.0 (TID 155) (localhost, executor driver, partition 0, NODE_LOCAL, 7860 bytes)
17:38:03.483 INFO TaskSetManager - Starting task 1.0 in stage 105.0 (TID 156) (localhost, executor driver, partition 1, NODE_LOCAL, 7860 bytes)
17:38:03.483 INFO Executor - Running task 0.0 in stage 105.0 (TID 155)
17:38:03.483 INFO Executor - Running task 1.0 in stage 105.0 (TID 156)
17:38:03.489 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:03.489 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:03.489 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:03.489 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:03.489 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:03.489 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:03.489 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:03.489 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:03.489 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:03.489 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:03.489 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:03.489 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:03.496 WARN IntelDeflaterFactory - Intel Deflater not supported, using Java.util.zip.Deflater
17:38:03.497 INFO ShuffleBlockFetcherIterator - Getting 1 (176.4 KiB) non-empty blocks including 1 (176.4 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
17:38:03.497 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
17:38:03.499 WARN IntelDeflaterFactory - Intel Deflater not supported, using Java.util.zip.Deflater
17:38:03.499 INFO ShuffleBlockFetcherIterator - Getting 1 (160.4 KiB) non-empty blocks including 1 (160.4 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
17:38:03.499 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
17:38:03.505 INFO FileOutputCommitter - Saved output of task 'attempt_202510271738033189862412434421744_0470_r_000001_0' to file:/tmp/ReadsSparkSinkUnitTest3524248484891532082.bam/_temporary/0/task_202510271738033189862412434421744_0470_r_000001
17:38:03.505 INFO SparkHadoopMapRedUtil - attempt_202510271738033189862412434421744_0470_r_000001_0: Committed. Elapsed time: 0 ms.
17:38:03.506 INFO Executor - Finished task 1.0 in stage 105.0 (TID 156). 1729 bytes result sent to driver
17:38:03.506 INFO TaskSetManager - Finished task 1.0 in stage 105.0 (TID 156) in 23 ms on localhost (executor driver) (1/2)
17:38:03.509 INFO FileOutputCommitter - Saved output of task 'attempt_202510271738033189862412434421744_0470_r_000000_0' to file:/tmp/ReadsSparkSinkUnitTest3524248484891532082.bam/_temporary/0/task_202510271738033189862412434421744_0470_r_000000
17:38:03.509 INFO SparkHadoopMapRedUtil - attempt_202510271738033189862412434421744_0470_r_000000_0: Committed. Elapsed time: 0 ms.
17:38:03.513 INFO Executor - Finished task 0.0 in stage 105.0 (TID 155). 1815 bytes result sent to driver
17:38:03.514 INFO BlockManagerInfo - Removed broadcast_185_piece0 on localhost:37953 in memory (size: 153.6 KiB, free: 1919.6 MiB)
17:38:03.514 INFO TaskSetManager - Finished task 0.0 in stage 105.0 (TID 155) in 32 ms on localhost (executor driver) (2/2)
17:38:03.514 INFO TaskSchedulerImpl - Removed TaskSet 105.0, whose tasks have all completed, from pool
17:38:03.514 INFO BlockManagerInfo - Removed broadcast_187_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.6 MiB)
17:38:03.514 INFO DAGScheduler - ResultStage 105 (runJob at SparkHadoopWriter.scala:83) finished in 0.039 s
17:38:03.514 INFO DAGScheduler - Job 74 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:03.514 INFO TaskSchedulerImpl - Killing all running tasks in stage 105: Stage finished
17:38:03.514 INFO DAGScheduler - Job 74 finished: runJob at SparkHadoopWriter.scala:83, took 0.112214 s
17:38:03.515 INFO SparkHadoopWriter - Start to commit write Job job_202510271738033189862412434421744_0470.
17:38:03.515 INFO BlockManagerInfo - Removed broadcast_177_piece0 on localhost:37953 in memory (size: 50.3 KiB, free: 1919.7 MiB)
17:38:03.516 INFO BlockManagerInfo - Removed broadcast_183_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.7 MiB)
17:38:03.518 INFO BlockManagerInfo - Removed broadcast_190_piece0 on localhost:37953 in memory (size: 154.6 KiB, free: 1919.9 MiB)
17:38:03.520 INFO SparkHadoopWriter - Write Job job_202510271738033189862412434421744_0470 committed. Elapsed time: 4 ms.
17:38:03.521 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:03.523 INFO MemoryStore - Block broadcast_192 stored as values in memory (estimated size 297.9 KiB, free 1919.1 MiB)
17:38:03.528 INFO MemoryStore - Block broadcast_192_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1919.1 MiB)
17:38:03.528 INFO BlockManagerInfo - Added broadcast_192_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.8 MiB)
17:38:03.529 INFO SparkContext - Created broadcast 192 from newAPIHadoopFile at PathSplitSource.java:96
17:38:03.551 INFO SparkContext - Starting job: count at ReadsSparkSinkUnitTest.java:222
17:38:03.551 INFO DAGScheduler - Got job 75 (count at ReadsSparkSinkUnitTest.java:222) with 2 output partitions
17:38:03.551 INFO DAGScheduler - Final stage: ResultStage 107 (count at ReadsSparkSinkUnitTest.java:222)
17:38:03.551 INFO DAGScheduler - Parents of final stage: List(ShuffleMapStage 106)
17:38:03.551 INFO DAGScheduler - Missing parents: List()
17:38:03.551 INFO DAGScheduler - Submitting ResultStage 107 (MapPartitionsRDD[461] at repartition at ReadsSparkSinkUnitTest.java:210), which has no missing parents
17:38:03.552 INFO MemoryStore - Block broadcast_193 stored as values in memory (estimated size 6.3 KiB, free 1919.1 MiB)
17:38:03.553 INFO MemoryStore - Block broadcast_193_piece0 stored as bytes in memory (estimated size 3.4 KiB, free 1919.1 MiB)
17:38:03.553 INFO BlockManagerInfo - Added broadcast_193_piece0 in memory on localhost:37953 (size: 3.4 KiB, free: 1919.8 MiB)
17:38:03.553 INFO SparkContext - Created broadcast 193 from broadcast at DAGScheduler.scala:1580
17:38:03.553 INFO DAGScheduler - Submitting 2 missing tasks from ResultStage 107 (MapPartitionsRDD[461] at repartition at ReadsSparkSinkUnitTest.java:210) (first 15 tasks are for partitions Vector(0, 1))
17:38:03.553 INFO TaskSchedulerImpl - Adding task set 107.0 with 2 tasks resource profile 0
17:38:03.554 INFO TaskSetManager - Starting task 0.0 in stage 107.0 (TID 157) (localhost, executor driver, partition 0, NODE_LOCAL, 7860 bytes)
17:38:03.554 INFO TaskSetManager - Starting task 1.0 in stage 107.0 (TID 158) (localhost, executor driver, partition 1, NODE_LOCAL, 7860 bytes)
17:38:03.555 INFO Executor - Running task 0.0 in stage 107.0 (TID 157)
17:38:03.556 INFO ShuffleBlockFetcherIterator - Getting 1 (160.4 KiB) non-empty blocks including 1 (160.4 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
17:38:03.556 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
17:38:03.557 INFO Executor - Running task 1.0 in stage 107.0 (TID 158)
17:38:03.558 INFO ShuffleBlockFetcherIterator - Getting 1 (176.4 KiB) non-empty blocks including 1 (176.4 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
17:38:03.558 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
17:38:03.559 INFO Executor - Finished task 0.0 in stage 107.0 (TID 157). 1591 bytes result sent to driver
17:38:03.560 INFO TaskSetManager - Finished task 0.0 in stage 107.0 (TID 157) in 6 ms on localhost (executor driver) (1/2)
17:38:03.561 INFO Executor - Finished task 1.0 in stage 107.0 (TID 158). 1591 bytes result sent to driver
17:38:03.561 INFO TaskSetManager - Finished task 1.0 in stage 107.0 (TID 158) in 7 ms on localhost (executor driver) (2/2)
17:38:03.561 INFO TaskSchedulerImpl - Removed TaskSet 107.0, whose tasks have all completed, from pool
17:38:03.561 INFO DAGScheduler - ResultStage 107 (count at ReadsSparkSinkUnitTest.java:222) finished in 0.009 s
17:38:03.561 INFO DAGScheduler - Job 75 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:03.561 INFO TaskSchedulerImpl - Killing all running tasks in stage 107: Stage finished
17:38:03.562 INFO DAGScheduler - Job 75 finished: count at ReadsSparkSinkUnitTest.java:222, took 0.010913 s
17:38:03.571 INFO FileInputFormat - Total input files to process : 2
17:38:03.574 INFO SparkContext - Starting job: count at ReadsSparkSinkUnitTest.java:222
17:38:03.574 INFO DAGScheduler - Got job 76 (count at ReadsSparkSinkUnitTest.java:222) with 2 output partitions
17:38:03.574 INFO DAGScheduler - Final stage: ResultStage 108 (count at ReadsSparkSinkUnitTest.java:222)
17:38:03.574 INFO DAGScheduler - Parents of final stage: List()
17:38:03.574 INFO DAGScheduler - Missing parents: List()
17:38:03.575 INFO DAGScheduler - Submitting ResultStage 108 (MapPartitionsRDD[477] at filter at ReadsSparkSource.java:96), which has no missing parents
17:38:03.591 INFO MemoryStore - Block broadcast_194 stored as values in memory (estimated size 426.1 KiB, free 1918.7 MiB)
17:38:03.592 INFO MemoryStore - Block broadcast_194_piece0 stored as bytes in memory (estimated size 153.6 KiB, free 1918.5 MiB)
17:38:03.592 INFO BlockManagerInfo - Added broadcast_194_piece0 in memory on localhost:37953 (size: 153.6 KiB, free: 1919.7 MiB)
17:38:03.593 INFO SparkContext - Created broadcast 194 from broadcast at DAGScheduler.scala:1580
17:38:03.593 INFO DAGScheduler - Submitting 2 missing tasks from ResultStage 108 (MapPartitionsRDD[477] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0, 1))
17:38:03.593 INFO TaskSchedulerImpl - Adding task set 108.0 with 2 tasks resource profile 0
17:38:03.594 INFO TaskSetManager - Starting task 0.0 in stage 108.0 (TID 159) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7825 bytes)
17:38:03.594 INFO TaskSetManager - Starting task 1.0 in stage 108.0 (TID 160) (localhost, executor driver, partition 1, PROCESS_LOCAL, 7825 bytes)
17:38:03.594 INFO Executor - Running task 1.0 in stage 108.0 (TID 160)
17:38:03.594 INFO Executor - Running task 0.0 in stage 108.0 (TID 159)
17:38:03.632 INFO NewHadoopRDD - Input split: file:/tmp/ReadsSparkSinkUnitTest3524248484891532082.bam/part-r-00001.bam:0+123180
17:38:03.632 INFO NewHadoopRDD - Input split: file:/tmp/ReadsSparkSinkUnitTest3524248484891532082.bam/part-r-00000.bam:0+122149
17:38:03.633 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:03.633 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:03.633 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:03.633 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:03.634 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:03.634 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:03.636 INFO Executor - Finished task 1.0 in stage 108.0 (TID 160). 989 bytes result sent to driver
17:38:03.636 INFO Executor - Finished task 0.0 in stage 108.0 (TID 159). 989 bytes result sent to driver
17:38:03.637 INFO TaskSetManager - Finished task 1.0 in stage 108.0 (TID 160) in 43 ms on localhost (executor driver) (1/2)
17:38:03.637 INFO TaskSetManager - Finished task 0.0 in stage 108.0 (TID 159) in 44 ms on localhost (executor driver) (2/2)
17:38:03.637 INFO TaskSchedulerImpl - Removed TaskSet 108.0, whose tasks have all completed, from pool
17:38:03.637 INFO DAGScheduler - ResultStage 108 (count at ReadsSparkSinkUnitTest.java:222) finished in 0.062 s
17:38:03.637 INFO DAGScheduler - Job 76 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:03.637 INFO TaskSchedulerImpl - Killing all running tasks in stage 108: Stage finished
17:38:03.637 INFO DAGScheduler - Job 76 finished: count at ReadsSparkSinkUnitTest.java:222, took 0.063499 s
17:38:03.641 INFO MemoryStore - Block broadcast_195 stored as values in memory (estimated size 576.0 B, free 1918.5 MiB)
17:38:03.641 INFO MemoryStore - Block broadcast_195_piece0 stored as bytes in memory (estimated size 228.0 B, free 1918.5 MiB)
17:38:03.641 INFO BlockManagerInfo - Added broadcast_195_piece0 in memory on localhost:37953 (size: 228.0 B, free: 1919.7 MiB)
17:38:03.642 INFO SparkContext - Created broadcast 195 from broadcast at CramSource.java:114
17:38:03.643 INFO MemoryStore - Block broadcast_196 stored as values in memory (estimated size 297.9 KiB, free 1918.2 MiB)
17:38:03.648 INFO MemoryStore - Block broadcast_196_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1918.2 MiB)
17:38:03.649 INFO BlockManagerInfo - Added broadcast_196_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.6 MiB)
17:38:03.649 INFO SparkContext - Created broadcast 196 from newAPIHadoopFile at PathSplitSource.java:96
17:38:03.666 INFO MemoryStore - Block broadcast_197 stored as values in memory (estimated size 576.0 B, free 1918.2 MiB)
17:38:03.666 INFO MemoryStore - Block broadcast_197_piece0 stored as bytes in memory (estimated size 228.0 B, free 1918.2 MiB)
17:38:03.667 INFO BlockManagerInfo - Added broadcast_197_piece0 in memory on localhost:37953 (size: 228.0 B, free: 1919.6 MiB)
17:38:03.667 INFO SparkContext - Created broadcast 197 from broadcast at CramSource.java:114
17:38:03.668 INFO MemoryStore - Block broadcast_198 stored as values in memory (estimated size 297.9 KiB, free 1917.9 MiB)
17:38:03.673 INFO MemoryStore - Block broadcast_198_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1917.8 MiB)
17:38:03.674 INFO BlockManagerInfo - Added broadcast_198_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.6 MiB)
17:38:03.674 INFO SparkContext - Created broadcast 198 from newAPIHadoopFile at PathSplitSource.java:96
17:38:03.687 INFO MemoryStore - Block broadcast_199 stored as values in memory (estimated size 6.0 KiB, free 1917.8 MiB)
17:38:03.690 INFO MemoryStore - Block broadcast_199_piece0 stored as bytes in memory (estimated size 1473.0 B, free 1917.8 MiB)
17:38:03.691 INFO BlockManagerInfo - Added broadcast_199_piece0 in memory on localhost:37953 (size: 1473.0 B, free: 1919.6 MiB)
17:38:03.691 INFO BlockManagerInfo - Removed broadcast_191_piece0 on localhost:37953 in memory (size: 56.2 KiB, free: 1919.6 MiB)
17:38:03.691 INFO SparkContext - Created broadcast 199 from broadcast at ReadsSparkSink.java:133
17:38:03.691 INFO BlockManagerInfo - Removed broadcast_186_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.7 MiB)
17:38:03.692 INFO MemoryStore - Block broadcast_200 stored as values in memory (estimated size 6.2 KiB, free 1918.3 MiB)
17:38:03.692 INFO MemoryStore - Block broadcast_200_piece0 stored as bytes in memory (estimated size 1473.0 B, free 1918.6 MiB)
17:38:03.692 INFO BlockManagerInfo - Added broadcast_200_piece0 in memory on localhost:37953 (size: 1473.0 B, free: 1919.7 MiB)
17:38:03.692 INFO BlockManagerInfo - Removed broadcast_192_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.7 MiB)
17:38:03.693 INFO SparkContext - Created broadcast 200 from broadcast at AnySamSinkMultiple.java:80
17:38:03.693 INFO BlockManagerInfo - Removed broadcast_188_piece0 on localhost:37953 in memory (size: 1890.0 B, free: 1919.7 MiB)
17:38:03.693 INFO BlockManagerInfo - Removed broadcast_198_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.8 MiB)
17:38:03.694 INFO BlockManagerInfo - Removed broadcast_189_piece0 on localhost:37953 in memory (size: 1890.0 B, free: 1919.8 MiB)
17:38:03.695 INFO BlockManagerInfo - Removed broadcast_194_piece0 on localhost:37953 in memory (size: 153.6 KiB, free: 1919.9 MiB)
17:38:03.695 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:03.695 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:03.695 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:03.695 INFO BlockManagerInfo - Removed broadcast_197_piece0 on localhost:37953 in memory (size: 228.0 B, free: 1919.9 MiB)
17:38:03.696 INFO BlockManagerInfo - Removed broadcast_193_piece0 on localhost:37953 in memory (size: 3.4 KiB, free: 1919.9 MiB)
17:38:03.704 INFO FileInputFormat - Total input files to process : 1
17:38:03.710 INFO SparkContext - Starting job: runJob at SparkHadoopWriter.scala:83
17:38:03.711 INFO DAGScheduler - Registering RDD 484 (repartition at ReadsSparkSinkUnitTest.java:210) as input to shuffle 23
17:38:03.711 INFO DAGScheduler - Got job 77 (runJob at SparkHadoopWriter.scala:83) with 2 output partitions
17:38:03.711 INFO DAGScheduler - Final stage: ResultStage 110 (runJob at SparkHadoopWriter.scala:83)
17:38:03.711 INFO DAGScheduler - Parents of final stage: List(ShuffleMapStage 109)
17:38:03.711 INFO DAGScheduler - Missing parents: List(ShuffleMapStage 109)
17:38:03.711 INFO DAGScheduler - Submitting ShuffleMapStage 109 (MapPartitionsRDD[484] at repartition at ReadsSparkSinkUnitTest.java:210), which has no missing parents
17:38:03.722 INFO MemoryStore - Block broadcast_201 stored as values in memory (estimated size 288.4 KiB, free 1919.4 MiB)
17:38:03.723 INFO MemoryStore - Block broadcast_201_piece0 stored as bytes in memory (estimated size 104.7 KiB, free 1919.3 MiB)
17:38:03.723 INFO BlockManagerInfo - Added broadcast_201_piece0 in memory on localhost:37953 (size: 104.7 KiB, free: 1919.8 MiB)
17:38:03.724 INFO SparkContext - Created broadcast 201 from broadcast at DAGScheduler.scala:1580
17:38:03.724 INFO DAGScheduler - Submitting 1 missing tasks from ShuffleMapStage 109 (MapPartitionsRDD[484] at repartition at ReadsSparkSinkUnitTest.java:210) (first 15 tasks are for partitions Vector(0))
17:38:03.724 INFO TaskSchedulerImpl - Adding task set 109.0 with 1 tasks resource profile 0
17:38:03.725 INFO TaskSetManager - Starting task 0.0 in stage 109.0 (TID 161) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7869 bytes)
17:38:03.725 INFO Executor - Running task 0.0 in stage 109.0 (TID 161)
17:38:03.748 INFO NewHadoopRDD - Input split: file:/home/runner/work/gatk/gatk/src/test/resources/org/broadinstitute/hellbender/tools/BQSR/NA12878.chr17_69k_70k.dictFix.cram:0+50619
17:38:03.761 INFO Executor - Finished task 0.0 in stage 109.0 (TID 161). 1149 bytes result sent to driver
17:38:03.762 INFO TaskSetManager - Finished task 0.0 in stage 109.0 (TID 161) in 38 ms on localhost (executor driver) (1/1)
17:38:03.762 INFO TaskSchedulerImpl - Removed TaskSet 109.0, whose tasks have all completed, from pool
17:38:03.762 INFO DAGScheduler - ShuffleMapStage 109 (repartition at ReadsSparkSinkUnitTest.java:210) finished in 0.051 s
17:38:03.762 INFO DAGScheduler - looking for newly runnable stages
17:38:03.762 INFO DAGScheduler - running: HashSet()
17:38:03.762 INFO DAGScheduler - waiting: HashSet(ResultStage 110)
17:38:03.762 INFO DAGScheduler - failed: HashSet()
17:38:03.762 INFO DAGScheduler - Submitting ResultStage 110 (MapPartitionsRDD[495] at mapToPair at AnySamSinkMultiple.java:89), which has no missing parents
17:38:03.768 INFO MemoryStore - Block broadcast_202 stored as values in memory (estimated size 150.3 KiB, free 1919.1 MiB)
17:38:03.769 INFO MemoryStore - Block broadcast_202_piece0 stored as bytes in memory (estimated size 56.4 KiB, free 1919.1 MiB)
17:38:03.769 INFO BlockManagerInfo - Added broadcast_202_piece0 in memory on localhost:37953 (size: 56.4 KiB, free: 1919.8 MiB)
17:38:03.769 INFO SparkContext - Created broadcast 202 from broadcast at DAGScheduler.scala:1580
17:38:03.770 INFO DAGScheduler - Submitting 2 missing tasks from ResultStage 110 (MapPartitionsRDD[495] at mapToPair at AnySamSinkMultiple.java:89) (first 15 tasks are for partitions Vector(0, 1))
17:38:03.770 INFO TaskSchedulerImpl - Adding task set 110.0 with 2 tasks resource profile 0
17:38:03.770 INFO TaskSetManager - Starting task 0.0 in stage 110.0 (TID 162) (localhost, executor driver, partition 0, NODE_LOCAL, 7860 bytes)
17:38:03.770 INFO TaskSetManager - Starting task 1.0 in stage 110.0 (TID 163) (localhost, executor driver, partition 1, NODE_LOCAL, 7860 bytes)
17:38:03.771 INFO Executor - Running task 0.0 in stage 110.0 (TID 162)
17:38:03.771 INFO Executor - Running task 1.0 in stage 110.0 (TID 163)
17:38:03.776 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:03.776 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:03.776 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:03.776 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:03.776 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:03.776 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:03.777 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:03.777 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:03.777 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:03.777 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:03.777 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:03.777 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:03.783 WARN IntelDeflaterFactory - Intel Deflater not supported, using Java.util.zip.Deflater
17:38:03.784 INFO ShuffleBlockFetcherIterator - Getting 1 (42.2 KiB) non-empty blocks including 1 (42.2 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
17:38:03.784 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
17:38:03.785 WARN IntelDeflaterFactory - Intel Deflater not supported, using Java.util.zip.Deflater
17:38:03.785 INFO ShuffleBlockFetcherIterator - Getting 1 (42.2 KiB) non-empty blocks including 1 (42.2 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
17:38:03.786 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
17:38:03.788 INFO FileOutputCommitter - Saved output of task 'attempt_202510271738035273547043722527462_0495_r_000000_0' to file:/tmp/ReadsSparkSinkUnitTest55583578263323576094.cram/_temporary/0/task_202510271738035273547043722527462_0495_r_000000
17:38:03.788 INFO SparkHadoopMapRedUtil - attempt_202510271738035273547043722527462_0495_r_000000_0: Committed. Elapsed time: 0 ms.
17:38:03.789 INFO Executor - Finished task 0.0 in stage 110.0 (TID 162). 1729 bytes result sent to driver
17:38:03.789 INFO TaskSetManager - Finished task 0.0 in stage 110.0 (TID 162) in 19 ms on localhost (executor driver) (1/2)
17:38:03.789 INFO FileOutputCommitter - Saved output of task 'attempt_202510271738035273547043722527462_0495_r_000001_0' to file:/tmp/ReadsSparkSinkUnitTest55583578263323576094.cram/_temporary/0/task_202510271738035273547043722527462_0495_r_000001
17:38:03.789 INFO SparkHadoopMapRedUtil - attempt_202510271738035273547043722527462_0495_r_000001_0: Committed. Elapsed time: 0 ms.
17:38:03.790 INFO Executor - Finished task 1.0 in stage 110.0 (TID 163). 1729 bytes result sent to driver
17:38:03.790 INFO TaskSetManager - Finished task 1.0 in stage 110.0 (TID 163) in 20 ms on localhost (executor driver) (2/2)
17:38:03.790 INFO TaskSchedulerImpl - Removed TaskSet 110.0, whose tasks have all completed, from pool
17:38:03.790 INFO DAGScheduler - ResultStage 110 (runJob at SparkHadoopWriter.scala:83) finished in 0.027 s
17:38:03.790 INFO DAGScheduler - Job 77 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:03.790 INFO TaskSchedulerImpl - Killing all running tasks in stage 110: Stage finished
17:38:03.790 INFO DAGScheduler - Job 77 finished: runJob at SparkHadoopWriter.scala:83, took 0.080019 s
17:38:03.790 INFO SparkHadoopWriter - Start to commit write Job job_202510271738035273547043722527462_0495.
17:38:03.795 INFO SparkHadoopWriter - Write Job job_202510271738035273547043722527462_0495 committed. Elapsed time: 4 ms.
17:38:03.796 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:03.797 INFO MemoryStore - Block broadcast_203 stored as values in memory (estimated size 297.9 KiB, free 1918.8 MiB)
17:38:03.803 INFO MemoryStore - Block broadcast_203_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1918.7 MiB)
17:38:03.803 INFO BlockManagerInfo - Added broadcast_203_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.7 MiB)
17:38:03.804 INFO SparkContext - Created broadcast 203 from newAPIHadoopFile at PathSplitSource.java:96
17:38:03.825 INFO SparkContext - Starting job: count at ReadsSparkSinkUnitTest.java:222
17:38:03.826 INFO DAGScheduler - Got job 78 (count at ReadsSparkSinkUnitTest.java:222) with 2 output partitions
17:38:03.826 INFO DAGScheduler - Final stage: ResultStage 112 (count at ReadsSparkSinkUnitTest.java:222)
17:38:03.826 INFO DAGScheduler - Parents of final stage: List(ShuffleMapStage 111)
17:38:03.826 INFO DAGScheduler - Missing parents: List()
17:38:03.826 INFO DAGScheduler - Submitting ResultStage 112 (MapPartitionsRDD[487] at repartition at ReadsSparkSinkUnitTest.java:210), which has no missing parents
17:38:03.827 INFO MemoryStore - Block broadcast_204 stored as values in memory (estimated size 6.3 KiB, free 1918.7 MiB)
17:38:03.827 INFO MemoryStore - Block broadcast_204_piece0 stored as bytes in memory (estimated size 3.4 KiB, free 1918.7 MiB)
17:38:03.827 INFO BlockManagerInfo - Added broadcast_204_piece0 in memory on localhost:37953 (size: 3.4 KiB, free: 1919.7 MiB)
17:38:03.828 INFO SparkContext - Created broadcast 204 from broadcast at DAGScheduler.scala:1580
17:38:03.828 INFO DAGScheduler - Submitting 2 missing tasks from ResultStage 112 (MapPartitionsRDD[487] at repartition at ReadsSparkSinkUnitTest.java:210) (first 15 tasks are for partitions Vector(0, 1))
17:38:03.828 INFO TaskSchedulerImpl - Adding task set 112.0 with 2 tasks resource profile 0
17:38:03.829 INFO TaskSetManager - Starting task 0.0 in stage 112.0 (TID 164) (localhost, executor driver, partition 0, NODE_LOCAL, 7860 bytes)
17:38:03.829 INFO TaskSetManager - Starting task 1.0 in stage 112.0 (TID 165) (localhost, executor driver, partition 1, NODE_LOCAL, 7860 bytes)
17:38:03.829 INFO Executor - Running task 0.0 in stage 112.0 (TID 164)
17:38:03.829 INFO Executor - Running task 1.0 in stage 112.0 (TID 165)
17:38:03.831 INFO ShuffleBlockFetcherIterator - Getting 1 (42.2 KiB) non-empty blocks including 1 (42.2 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
17:38:03.831 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
17:38:03.831 INFO ShuffleBlockFetcherIterator - Getting 1 (42.2 KiB) non-empty blocks including 1 (42.2 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
17:38:03.831 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
17:38:03.833 INFO Executor - Finished task 1.0 in stage 112.0 (TID 165). 1591 bytes result sent to driver
17:38:03.833 INFO Executor - Finished task 0.0 in stage 112.0 (TID 164). 1591 bytes result sent to driver
17:38:03.834 INFO TaskSetManager - Finished task 1.0 in stage 112.0 (TID 165) in 5 ms on localhost (executor driver) (1/2)
17:38:03.834 INFO TaskSetManager - Finished task 0.0 in stage 112.0 (TID 164) in 6 ms on localhost (executor driver) (2/2)
17:38:03.834 INFO TaskSchedulerImpl - Removed TaskSet 112.0, whose tasks have all completed, from pool
17:38:03.834 INFO DAGScheduler - ResultStage 112 (count at ReadsSparkSinkUnitTest.java:222) finished in 0.007 s
17:38:03.834 INFO DAGScheduler - Job 78 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:03.834 INFO TaskSchedulerImpl - Killing all running tasks in stage 112: Stage finished
17:38:03.834 INFO DAGScheduler - Job 78 finished: count at ReadsSparkSinkUnitTest.java:222, took 0.009069 s
17:38:03.843 INFO FileInputFormat - Total input files to process : 2
17:38:03.847 INFO SparkContext - Starting job: count at ReadsSparkSinkUnitTest.java:222
17:38:03.847 INFO DAGScheduler - Got job 79 (count at ReadsSparkSinkUnitTest.java:222) with 2 output partitions
17:38:03.847 INFO DAGScheduler - Final stage: ResultStage 113 (count at ReadsSparkSinkUnitTest.java:222)
17:38:03.847 INFO DAGScheduler - Parents of final stage: List()
17:38:03.847 INFO DAGScheduler - Missing parents: List()
17:38:03.847 INFO DAGScheduler - Submitting ResultStage 113 (MapPartitionsRDD[502] at filter at ReadsSparkSource.java:96), which has no missing parents
17:38:03.863 INFO MemoryStore - Block broadcast_205 stored as values in memory (estimated size 426.1 KiB, free 1918.3 MiB)
17:38:03.864 INFO MemoryStore - Block broadcast_205_piece0 stored as bytes in memory (estimated size 153.6 KiB, free 1918.1 MiB)
17:38:03.864 INFO BlockManagerInfo - Added broadcast_205_piece0 in memory on localhost:37953 (size: 153.6 KiB, free: 1919.6 MiB)
17:38:03.865 INFO SparkContext - Created broadcast 205 from broadcast at DAGScheduler.scala:1580
17:38:03.865 INFO DAGScheduler - Submitting 2 missing tasks from ResultStage 113 (MapPartitionsRDD[502] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0, 1))
17:38:03.865 INFO TaskSchedulerImpl - Adding task set 113.0 with 2 tasks resource profile 0
17:38:03.866 INFO TaskSetManager - Starting task 0.0 in stage 113.0 (TID 166) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7827 bytes)
17:38:03.866 INFO TaskSetManager - Starting task 1.0 in stage 113.0 (TID 167) (localhost, executor driver, partition 1, PROCESS_LOCAL, 7827 bytes)
17:38:03.866 INFO Executor - Running task 0.0 in stage 113.0 (TID 166)
17:38:03.866 INFO Executor - Running task 1.0 in stage 113.0 (TID 167)
17:38:03.904 INFO NewHadoopRDD - Input split: file:/tmp/ReadsSparkSinkUnitTest55583578263323576094.cram/part-r-00000.bam:0+31767
17:38:03.904 INFO NewHadoopRDD - Input split: file:/tmp/ReadsSparkSinkUnitTest55583578263323576094.cram/part-r-00001.bam:0+31081
17:38:03.905 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:03.905 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:03.906 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:03.906 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:03.906 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:03.906 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:03.911 INFO Executor - Finished task 1.0 in stage 113.0 (TID 167). 1075 bytes result sent to driver
17:38:03.911 INFO Executor - Finished task 0.0 in stage 113.0 (TID 166). 1075 bytes result sent to driver
17:38:03.911 INFO BlockManagerInfo - Removed broadcast_200_piece0 on localhost:37953 in memory (size: 1473.0 B, free: 1919.6 MiB)
17:38:03.911 INFO TaskSetManager - Finished task 1.0 in stage 113.0 (TID 167) in 45 ms on localhost (executor driver) (1/2)
17:38:03.912 INFO TaskSetManager - Finished task 0.0 in stage 113.0 (TID 166) in 46 ms on localhost (executor driver) (2/2)
17:38:03.912 INFO TaskSchedulerImpl - Removed TaskSet 113.0, whose tasks have all completed, from pool
17:38:03.912 INFO DAGScheduler - ResultStage 113 (count at ReadsSparkSinkUnitTest.java:222) finished in 0.064 s
17:38:03.912 INFO DAGScheduler - Job 79 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:03.912 INFO TaskSchedulerImpl - Killing all running tasks in stage 113: Stage finished
17:38:03.912 INFO DAGScheduler - Job 79 finished: count at ReadsSparkSinkUnitTest.java:222, took 0.065464 s
17:38:03.912 INFO BlockManagerInfo - Removed broadcast_201_piece0 on localhost:37953 in memory (size: 104.7 KiB, free: 1919.7 MiB)
17:38:03.913 INFO BlockManagerInfo - Removed broadcast_204_piece0 on localhost:37953 in memory (size: 3.4 KiB, free: 1919.7 MiB)
17:38:03.913 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:03.914 INFO BlockManagerInfo - Removed broadcast_202_piece0 on localhost:37953 in memory (size: 56.4 KiB, free: 1919.8 MiB)
17:38:03.914 INFO BlockManagerInfo - Removed broadcast_199_piece0 on localhost:37953 in memory (size: 1473.0 B, free: 1919.8 MiB)
17:38:03.916 INFO MemoryStore - Block broadcast_206 stored as values in memory (estimated size 297.9 KiB, free 1918.5 MiB)
17:38:03.921 INFO MemoryStore - Block broadcast_206_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1918.4 MiB)
17:38:03.921 INFO BlockManagerInfo - Added broadcast_206_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.7 MiB)
17:38:03.922 INFO SparkContext - Created broadcast 206 from newAPIHadoopFile at PathSplitSource.java:96
17:38:03.943 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:03.945 INFO MemoryStore - Block broadcast_207 stored as values in memory (estimated size 297.9 KiB, free 1918.1 MiB)
17:38:03.951 INFO MemoryStore - Block broadcast_207_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1918.1 MiB)
17:38:03.951 INFO BlockManagerInfo - Added broadcast_207_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.7 MiB)
17:38:03.952 INFO SparkContext - Created broadcast 207 from newAPIHadoopFile at PathSplitSource.java:96
17:38:03.971 INFO MemoryStore - Block broadcast_208 stored as values in memory (estimated size 160.7 KiB, free 1917.9 MiB)
17:38:03.972 INFO MemoryStore - Block broadcast_208_piece0 stored as bytes in memory (estimated size 9.6 KiB, free 1917.9 MiB)
17:38:03.972 INFO BlockManagerInfo - Added broadcast_208_piece0 in memory on localhost:37953 (size: 9.6 KiB, free: 1919.6 MiB)
17:38:03.973 INFO SparkContext - Created broadcast 208 from broadcast at ReadsSparkSink.java:133
17:38:03.974 INFO MemoryStore - Block broadcast_209 stored as values in memory (estimated size 163.2 KiB, free 1917.7 MiB)
17:38:03.975 INFO MemoryStore - Block broadcast_209_piece0 stored as bytes in memory (estimated size 9.6 KiB, free 1917.7 MiB)
17:38:03.975 INFO BlockManagerInfo - Added broadcast_209_piece0 in memory on localhost:37953 (size: 9.6 KiB, free: 1919.6 MiB)
17:38:03.975 INFO SparkContext - Created broadcast 209 from broadcast at AnySamSinkMultiple.java:80
17:38:03.977 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:03.977 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:03.977 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:03.986 INFO FileInputFormat - Total input files to process : 1
17:38:03.992 INFO SparkContext - Starting job: runJob at SparkHadoopWriter.scala:83
17:38:03.992 INFO DAGScheduler - Registering RDD 510 (repartition at ReadsSparkSinkUnitTest.java:210) as input to shuffle 24
17:38:03.993 INFO DAGScheduler - Got job 80 (runJob at SparkHadoopWriter.scala:83) with 2 output partitions
17:38:03.993 INFO DAGScheduler - Final stage: ResultStage 115 (runJob at SparkHadoopWriter.scala:83)
17:38:03.993 INFO DAGScheduler - Parents of final stage: List(ShuffleMapStage 114)
17:38:03.993 INFO DAGScheduler - Missing parents: List(ShuffleMapStage 114)
17:38:03.993 INFO DAGScheduler - Submitting ShuffleMapStage 114 (MapPartitionsRDD[510] at repartition at ReadsSparkSinkUnitTest.java:210), which has no missing parents
17:38:04.008 INFO MemoryStore - Block broadcast_210 stored as values in memory (estimated size 427.7 KiB, free 1917.3 MiB)
17:38:04.009 INFO MemoryStore - Block broadcast_210_piece0 stored as bytes in memory (estimated size 154.6 KiB, free 1917.2 MiB)
17:38:04.010 INFO BlockManagerInfo - Added broadcast_210_piece0 in memory on localhost:37953 (size: 154.6 KiB, free: 1919.5 MiB)
17:38:04.010 INFO SparkContext - Created broadcast 210 from broadcast at DAGScheduler.scala:1580
17:38:04.010 INFO DAGScheduler - Submitting 1 missing tasks from ShuffleMapStage 114 (MapPartitionsRDD[510] at repartition at ReadsSparkSinkUnitTest.java:210) (first 15 tasks are for partitions Vector(0))
17:38:04.010 INFO TaskSchedulerImpl - Adding task set 114.0 with 1 tasks resource profile 0
17:38:04.011 INFO TaskSetManager - Starting task 0.0 in stage 114.0 (TID 168) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7861 bytes)
17:38:04.011 INFO Executor - Running task 0.0 in stage 114.0 (TID 168)
17:38:04.045 INFO NewHadoopRDD - Input split: file:/home/runner/work/gatk/gatk/src/test/resources/org/broadinstitute/hellbender/tools/BQSR/HiSeq.1mb.1RG.2k_lines.bam:0+222075
17:38:04.046 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:04.047 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:04.054 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:04.065 INFO Executor - Finished task 0.0 in stage 114.0 (TID 168). 1149 bytes result sent to driver
17:38:04.065 INFO TaskSetManager - Finished task 0.0 in stage 114.0 (TID 168) in 54 ms on localhost (executor driver) (1/1)
17:38:04.065 INFO TaskSchedulerImpl - Removed TaskSet 114.0, whose tasks have all completed, from pool
17:38:04.066 INFO DAGScheduler - ShuffleMapStage 114 (repartition at ReadsSparkSinkUnitTest.java:210) finished in 0.073 s
17:38:04.066 INFO DAGScheduler - looking for newly runnable stages
17:38:04.066 INFO DAGScheduler - running: HashSet()
17:38:04.066 INFO DAGScheduler - waiting: HashSet(ResultStage 115)
17:38:04.066 INFO DAGScheduler - failed: HashSet()
17:38:04.066 INFO DAGScheduler - Submitting ResultStage 115 (MapPartitionsRDD[522] at mapToPair at AnySamSinkMultiple.java:89), which has no missing parents
17:38:04.072 INFO MemoryStore - Block broadcast_211 stored as values in memory (estimated size 150.2 KiB, free 1917.0 MiB)
17:38:04.073 INFO MemoryStore - Block broadcast_211_piece0 stored as bytes in memory (estimated size 56.2 KiB, free 1917.0 MiB)
17:38:04.073 INFO BlockManagerInfo - Added broadcast_211_piece0 in memory on localhost:37953 (size: 56.2 KiB, free: 1919.4 MiB)
17:38:04.073 INFO SparkContext - Created broadcast 211 from broadcast at DAGScheduler.scala:1580
17:38:04.073 INFO DAGScheduler - Submitting 2 missing tasks from ResultStage 115 (MapPartitionsRDD[522] at mapToPair at AnySamSinkMultiple.java:89) (first 15 tasks are for partitions Vector(0, 1))
17:38:04.073 INFO TaskSchedulerImpl - Adding task set 115.0 with 2 tasks resource profile 0
17:38:04.074 INFO TaskSetManager - Starting task 0.0 in stage 115.0 (TID 169) (localhost, executor driver, partition 0, NODE_LOCAL, 7860 bytes)
17:38:04.074 INFO TaskSetManager - Starting task 1.0 in stage 115.0 (TID 170) (localhost, executor driver, partition 1, NODE_LOCAL, 7860 bytes)
17:38:04.074 INFO Executor - Running task 0.0 in stage 115.0 (TID 169)
17:38:04.075 INFO Executor - Running task 1.0 in stage 115.0 (TID 170)
17:38:04.080 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:04.080 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:04.080 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:04.080 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:04.080 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:04.080 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:04.080 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:04.080 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:04.080 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:04.080 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:04.080 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:04.080 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:04.087 WARN IntelDeflaterFactory - Intel Deflater not supported, using Java.util.zip.Deflater
17:38:04.088 INFO ShuffleBlockFetcherIterator - Getting 1 (176.4 KiB) non-empty blocks including 1 (176.4 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
17:38:04.088 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
17:38:04.090 WARN IntelDeflaterFactory - Intel Deflater not supported, using Java.util.zip.Deflater
17:38:04.091 INFO ShuffleBlockFetcherIterator - Getting 1 (176.4 KiB) non-empty blocks including 1 (176.4 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
17:38:04.091 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
17:38:04.099 INFO FileOutputCommitter - Saved output of task 'attempt_202510271738037392616294977839058_0522_r_000000_0' to file:/tmp/ReadsSparkSinkUnitTest614023005047082193516.sam/_temporary/0/task_202510271738037392616294977839058_0522_r_000000
17:38:04.099 INFO FileOutputCommitter - Saved output of task 'attempt_202510271738037392616294977839058_0522_r_000001_0' to file:/tmp/ReadsSparkSinkUnitTest614023005047082193516.sam/_temporary/0/task_202510271738037392616294977839058_0522_r_000001
17:38:04.099 INFO SparkHadoopMapRedUtil - attempt_202510271738037392616294977839058_0522_r_000000_0: Committed. Elapsed time: 0 ms.
17:38:04.099 INFO SparkHadoopMapRedUtil - attempt_202510271738037392616294977839058_0522_r_000001_0: Committed. Elapsed time: 0 ms.
17:38:04.099 INFO Executor - Finished task 0.0 in stage 115.0 (TID 169). 1729 bytes result sent to driver
17:38:04.099 INFO Executor - Finished task 1.0 in stage 115.0 (TID 170). 1729 bytes result sent to driver
17:38:04.100 INFO TaskSetManager - Finished task 1.0 in stage 115.0 (TID 170) in 26 ms on localhost (executor driver) (1/2)
17:38:04.100 INFO TaskSetManager - Finished task 0.0 in stage 115.0 (TID 169) in 26 ms on localhost (executor driver) (2/2)
17:38:04.100 INFO TaskSchedulerImpl - Removed TaskSet 115.0, whose tasks have all completed, from pool
17:38:04.100 INFO DAGScheduler - ResultStage 115 (runJob at SparkHadoopWriter.scala:83) finished in 0.034 s
17:38:04.100 INFO DAGScheduler - Job 80 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:04.100 INFO TaskSchedulerImpl - Killing all running tasks in stage 115: Stage finished
17:38:04.101 INFO DAGScheduler - Job 80 finished: runJob at SparkHadoopWriter.scala:83, took 0.108745 s
17:38:04.101 INFO SparkHadoopWriter - Start to commit write Job job_202510271738037392616294977839058_0522.
17:38:04.105 INFO SparkHadoopWriter - Write Job job_202510271738037392616294977839058_0522 committed. Elapsed time: 4 ms.
17:38:04.106 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:04.108 INFO MemoryStore - Block broadcast_212 stored as values in memory (estimated size 297.9 KiB, free 1916.7 MiB)
17:38:04.114 INFO MemoryStore - Block broadcast_212_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1916.6 MiB)
17:38:04.114 INFO BlockManagerInfo - Added broadcast_212_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.4 MiB)
17:38:04.114 INFO SparkContext - Created broadcast 212 from newAPIHadoopFile at PathSplitSource.java:96
17:38:04.136 INFO SparkContext - Starting job: count at ReadsSparkSinkUnitTest.java:222
17:38:04.137 INFO DAGScheduler - Got job 81 (count at ReadsSparkSinkUnitTest.java:222) with 2 output partitions
17:38:04.137 INFO DAGScheduler - Final stage: ResultStage 117 (count at ReadsSparkSinkUnitTest.java:222)
17:38:04.137 INFO DAGScheduler - Parents of final stage: List(ShuffleMapStage 116)
17:38:04.137 INFO DAGScheduler - Missing parents: List()
17:38:04.137 INFO DAGScheduler - Submitting ResultStage 117 (MapPartitionsRDD[513] at repartition at ReadsSparkSinkUnitTest.java:210), which has no missing parents
17:38:04.138 INFO MemoryStore - Block broadcast_213 stored as values in memory (estimated size 6.3 KiB, free 1916.6 MiB)
17:38:04.141 INFO MemoryStore - Block broadcast_213_piece0 stored as bytes in memory (estimated size 3.4 KiB, free 1916.6 MiB)
17:38:04.142 INFO BlockManagerInfo - Added broadcast_213_piece0 in memory on localhost:37953 (size: 3.4 KiB, free: 1919.4 MiB)
17:38:04.142 INFO BlockManagerInfo - Removed broadcast_208_piece0 on localhost:37953 in memory (size: 9.6 KiB, free: 1919.4 MiB)
17:38:04.142 INFO SparkContext - Created broadcast 213 from broadcast at DAGScheduler.scala:1580
17:38:04.142 INFO DAGScheduler - Submitting 2 missing tasks from ResultStage 117 (MapPartitionsRDD[513] at repartition at ReadsSparkSinkUnitTest.java:210) (first 15 tasks are for partitions Vector(0, 1))
17:38:04.142 INFO TaskSchedulerImpl - Adding task set 117.0 with 2 tasks resource profile 0
17:38:04.143 INFO TaskSetManager - Starting task 0.0 in stage 117.0 (TID 171) (localhost, executor driver, partition 0, NODE_LOCAL, 7860 bytes)
17:38:04.143 INFO BlockManagerInfo - Removed broadcast_207_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.4 MiB)
17:38:04.143 INFO TaskSetManager - Starting task 1.0 in stage 117.0 (TID 172) (localhost, executor driver, partition 1, NODE_LOCAL, 7860 bytes)
17:38:04.143 INFO Executor - Running task 0.0 in stage 117.0 (TID 171)
17:38:04.144 INFO Executor - Running task 1.0 in stage 117.0 (TID 172)
17:38:04.145 INFO ShuffleBlockFetcherIterator - Getting 1 (176.4 KiB) non-empty blocks including 1 (176.4 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
17:38:04.145 INFO ShuffleBlockFetcherIterator - Getting 1 (176.4 KiB) non-empty blocks including 1 (176.4 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
17:38:04.145 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
17:38:04.145 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
17:38:04.147 INFO BlockManagerInfo - Removed broadcast_209_piece0 on localhost:37953 in memory (size: 9.6 KiB, free: 1919.4 MiB)
17:38:04.147 INFO BlockManagerInfo - Removed broadcast_195_piece0 on localhost:37953 in memory (size: 228.0 B, free: 1919.4 MiB)
17:38:04.148 INFO BlockManagerInfo - Removed broadcast_205_piece0 on localhost:37953 in memory (size: 153.6 KiB, free: 1919.6 MiB)
17:38:04.148 INFO Executor - Finished task 1.0 in stage 117.0 (TID 172). 1591 bytes result sent to driver
17:38:04.149 INFO TaskSetManager - Finished task 1.0 in stage 117.0 (TID 172) in 6 ms on localhost (executor driver) (1/2)
17:38:04.149 INFO BlockManagerInfo - Removed broadcast_210_piece0 on localhost:37953 in memory (size: 154.6 KiB, free: 1919.7 MiB)
17:38:04.149 INFO Executor - Finished task 0.0 in stage 117.0 (TID 171). 1591 bytes result sent to driver
17:38:04.149 INFO TaskSetManager - Finished task 0.0 in stage 117.0 (TID 171) in 6 ms on localhost (executor driver) (2/2)
17:38:04.149 INFO TaskSchedulerImpl - Removed TaskSet 117.0, whose tasks have all completed, from pool
17:38:04.150 INFO DAGScheduler - ResultStage 117 (count at ReadsSparkSinkUnitTest.java:222) finished in 0.013 s
17:38:04.150 INFO BlockManagerInfo - Removed broadcast_203_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.8 MiB)
17:38:04.151 INFO DAGScheduler - Job 81 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:04.151 INFO TaskSchedulerImpl - Killing all running tasks in stage 117: Stage finished
17:38:04.151 INFO DAGScheduler - Job 81 finished: count at ReadsSparkSinkUnitTest.java:222, took 0.014609 s
17:38:04.151 INFO BlockManagerInfo - Removed broadcast_196_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.8 MiB)
17:38:04.152 INFO BlockManagerInfo - Removed broadcast_211_piece0 on localhost:37953 in memory (size: 56.2 KiB, free: 1919.9 MiB)
17:38:04.160 INFO FileInputFormat - Total input files to process : 2
17:38:04.163 INFO SparkContext - Starting job: count at ReadsSparkSinkUnitTest.java:222
17:38:04.163 INFO DAGScheduler - Got job 82 (count at ReadsSparkSinkUnitTest.java:222) with 2 output partitions
17:38:04.163 INFO DAGScheduler - Final stage: ResultStage 118 (count at ReadsSparkSinkUnitTest.java:222)
17:38:04.163 INFO DAGScheduler - Parents of final stage: List()
17:38:04.163 INFO DAGScheduler - Missing parents: List()
17:38:04.164 INFO DAGScheduler - Submitting ResultStage 118 (MapPartitionsRDD[529] at filter at ReadsSparkSource.java:96), which has no missing parents
17:38:04.179 INFO MemoryStore - Block broadcast_214 stored as values in memory (estimated size 426.1 KiB, free 1918.9 MiB)
17:38:04.181 INFO MemoryStore - Block broadcast_214_piece0 stored as bytes in memory (estimated size 153.6 KiB, free 1918.7 MiB)
17:38:04.181 INFO BlockManagerInfo - Added broadcast_214_piece0 in memory on localhost:37953 (size: 153.6 KiB, free: 1919.7 MiB)
17:38:04.181 INFO SparkContext - Created broadcast 214 from broadcast at DAGScheduler.scala:1580
17:38:04.181 INFO DAGScheduler - Submitting 2 missing tasks from ResultStage 118 (MapPartitionsRDD[529] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0, 1))
17:38:04.181 INFO TaskSchedulerImpl - Adding task set 118.0 with 2 tasks resource profile 0
17:38:04.182 INFO TaskSetManager - Starting task 0.0 in stage 118.0 (TID 173) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7827 bytes)
17:38:04.182 INFO TaskSetManager - Starting task 1.0 in stage 118.0 (TID 174) (localhost, executor driver, partition 1, PROCESS_LOCAL, 7827 bytes)
17:38:04.182 INFO Executor - Running task 1.0 in stage 118.0 (TID 174)
17:38:04.182 INFO Executor - Running task 0.0 in stage 118.0 (TID 173)
17:38:04.219 INFO NewHadoopRDD - Input split: file:/tmp/ReadsSparkSinkUnitTest614023005047082193516.sam/part-r-00000.bam:0+133545
17:38:04.219 INFO NewHadoopRDD - Input split: file:/tmp/ReadsSparkSinkUnitTest614023005047082193516.sam/part-r-00001.bam:0+129979
17:38:04.220 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:04.220 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:04.221 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:04.221 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:04.229 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:04.229 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:04.232 INFO Executor - Finished task 1.0 in stage 118.0 (TID 174). 989 bytes result sent to driver
17:38:04.232 INFO Executor - Finished task 0.0 in stage 118.0 (TID 173). 989 bytes result sent to driver
17:38:04.232 INFO TaskSetManager - Finished task 0.0 in stage 118.0 (TID 173) in 50 ms on localhost (executor driver) (1/2)
17:38:04.232 INFO TaskSetManager - Finished task 1.0 in stage 118.0 (TID 174) in 50 ms on localhost (executor driver) (2/2)
17:38:04.232 INFO TaskSchedulerImpl - Removed TaskSet 118.0, whose tasks have all completed, from pool
17:38:04.232 INFO DAGScheduler - ResultStage 118 (count at ReadsSparkSinkUnitTest.java:222) finished in 0.068 s
17:38:04.233 INFO DAGScheduler - Job 82 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:04.233 INFO TaskSchedulerImpl - Killing all running tasks in stage 118: Stage finished
17:38:04.233 INFO DAGScheduler - Job 82 finished: count at ReadsSparkSinkUnitTest.java:222, took 0.069744 s
17:38:04.234 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:04.237 INFO MemoryStore - Block broadcast_215 stored as values in memory (estimated size 297.9 KiB, free 1918.5 MiB)
17:38:04.242 INFO MemoryStore - Block broadcast_215_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1918.4 MiB)
17:38:04.243 INFO BlockManagerInfo - Added broadcast_215_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.7 MiB)
17:38:04.243 INFO SparkContext - Created broadcast 215 from newAPIHadoopFile at PathSplitSource.java:96
17:38:04.263 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:04.265 INFO MemoryStore - Block broadcast_216 stored as values in memory (estimated size 297.9 KiB, free 1918.1 MiB)
17:38:04.271 INFO MemoryStore - Block broadcast_216_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1918.1 MiB)
17:38:04.271 INFO BlockManagerInfo - Added broadcast_216_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.7 MiB)
17:38:04.271 INFO SparkContext - Created broadcast 216 from newAPIHadoopFile at PathSplitSource.java:96
17:38:04.291 INFO FileInputFormat - Total input files to process : 1
17:38:04.293 INFO MemoryStore - Block broadcast_217 stored as values in memory (estimated size 160.7 KiB, free 1917.9 MiB)
17:38:04.294 INFO MemoryStore - Block broadcast_217_piece0 stored as bytes in memory (estimated size 9.6 KiB, free 1917.9 MiB)
17:38:04.294 INFO BlockManagerInfo - Added broadcast_217_piece0 in memory on localhost:37953 (size: 9.6 KiB, free: 1919.6 MiB)
17:38:04.294 INFO SparkContext - Created broadcast 217 from broadcast at ReadsSparkSink.java:133
17:38:04.295 INFO MemoryStore - Block broadcast_218 stored as values in memory (estimated size 163.2 KiB, free 1917.7 MiB)
17:38:04.296 INFO MemoryStore - Block broadcast_218_piece0 stored as bytes in memory (estimated size 9.6 KiB, free 1917.7 MiB)
17:38:04.296 INFO BlockManagerInfo - Added broadcast_218_piece0 in memory on localhost:37953 (size: 9.6 KiB, free: 1919.6 MiB)
17:38:04.296 INFO SparkContext - Created broadcast 218 from broadcast at BamSink.java:76
17:38:04.298 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:04.299 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:04.299 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:04.312 INFO SparkContext - Starting job: runJob at SparkHadoopWriter.scala:83
17:38:04.313 INFO DAGScheduler - Registering RDD 543 (mapToPair at SparkUtils.java:161) as input to shuffle 25
17:38:04.313 INFO DAGScheduler - Got job 83 (runJob at SparkHadoopWriter.scala:83) with 1 output partitions
17:38:04.313 INFO DAGScheduler - Final stage: ResultStage 120 (runJob at SparkHadoopWriter.scala:83)
17:38:04.313 INFO DAGScheduler - Parents of final stage: List(ShuffleMapStage 119)
17:38:04.313 INFO DAGScheduler - Missing parents: List(ShuffleMapStage 119)
17:38:04.313 INFO DAGScheduler - Submitting ShuffleMapStage 119 (MapPartitionsRDD[543] at mapToPair at SparkUtils.java:161), which has no missing parents
17:38:04.330 INFO MemoryStore - Block broadcast_219 stored as values in memory (estimated size 520.4 KiB, free 1917.2 MiB)
17:38:04.331 INFO MemoryStore - Block broadcast_219_piece0 stored as bytes in memory (estimated size 166.1 KiB, free 1917.1 MiB)
17:38:04.331 INFO BlockManagerInfo - Added broadcast_219_piece0 in memory on localhost:37953 (size: 166.1 KiB, free: 1919.5 MiB)
17:38:04.331 INFO SparkContext - Created broadcast 219 from broadcast at DAGScheduler.scala:1580
17:38:04.332 INFO DAGScheduler - Submitting 1 missing tasks from ShuffleMapStage 119 (MapPartitionsRDD[543] at mapToPair at SparkUtils.java:161) (first 15 tasks are for partitions Vector(0))
17:38:04.332 INFO TaskSchedulerImpl - Adding task set 119.0 with 1 tasks resource profile 0
17:38:04.332 INFO TaskSetManager - Starting task 0.0 in stage 119.0 (TID 175) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7861 bytes)
17:38:04.333 INFO Executor - Running task 0.0 in stage 119.0 (TID 175)
17:38:04.367 INFO NewHadoopRDD - Input split: file:/home/runner/work/gatk/gatk/src/test/resources/org/broadinstitute/hellbender/tools/BQSR/HiSeq.1mb.1RG.2k_lines.bam:0+222075
17:38:04.368 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:04.369 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:04.376 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:04.389 INFO Executor - Finished task 0.0 in stage 119.0 (TID 175). 1234 bytes result sent to driver
17:38:04.390 INFO TaskSetManager - Finished task 0.0 in stage 119.0 (TID 175) in 58 ms on localhost (executor driver) (1/1)
17:38:04.390 INFO TaskSchedulerImpl - Removed TaskSet 119.0, whose tasks have all completed, from pool
17:38:04.390 INFO BlockManagerInfo - Removed broadcast_216_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.5 MiB)
17:38:04.390 INFO DAGScheduler - ShuffleMapStage 119 (mapToPair at SparkUtils.java:161) finished in 0.076 s
17:38:04.390 INFO DAGScheduler - looking for newly runnable stages
17:38:04.390 INFO DAGScheduler - running: HashSet()
17:38:04.390 INFO DAGScheduler - waiting: HashSet(ResultStage 120)
17:38:04.390 INFO DAGScheduler - failed: HashSet()
17:38:04.390 INFO DAGScheduler - Submitting ResultStage 120 (MapPartitionsRDD[548] at mapToPair at BamSink.java:91), which has no missing parents
17:38:04.390 INFO BlockManagerInfo - Removed broadcast_214_piece0 on localhost:37953 in memory (size: 153.6 KiB, free: 1919.7 MiB)
17:38:04.391 INFO BlockManagerInfo - Removed broadcast_212_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.7 MiB)
17:38:04.392 INFO BlockManagerInfo - Removed broadcast_213_piece0 on localhost:37953 in memory (size: 3.4 KiB, free: 1919.7 MiB)
17:38:04.392 INFO BlockManagerInfo - Removed broadcast_206_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.8 MiB)
17:38:04.397 INFO MemoryStore - Block broadcast_220 stored as values in memory (estimated size 241.4 KiB, free 1918.4 MiB)
17:38:04.397 INFO MemoryStore - Block broadcast_220_piece0 stored as bytes in memory (estimated size 67.0 KiB, free 1918.4 MiB)
17:38:04.397 INFO BlockManagerInfo - Added broadcast_220_piece0 in memory on localhost:37953 (size: 67.0 KiB, free: 1919.7 MiB)
17:38:04.398 INFO SparkContext - Created broadcast 220 from broadcast at DAGScheduler.scala:1580
17:38:04.398 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 120 (MapPartitionsRDD[548] at mapToPair at BamSink.java:91) (first 15 tasks are for partitions Vector(0))
17:38:04.398 INFO TaskSchedulerImpl - Adding task set 120.0 with 1 tasks resource profile 0
17:38:04.399 INFO TaskSetManager - Starting task 0.0 in stage 120.0 (TID 176) (localhost, executor driver, partition 0, NODE_LOCAL, 7513 bytes)
17:38:04.399 INFO Executor - Running task 0.0 in stage 120.0 (TID 176)
17:38:04.405 INFO ShuffleBlockFetcherIterator - Getting 1 (343.8 KiB) non-empty blocks including 1 (343.8 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
17:38:04.405 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
17:38:04.418 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:04.418 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:04.418 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:04.419 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:04.419 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:04.419 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:04.427 WARN IntelDeflaterFactory - Intel Deflater not supported, using Java.util.zip.Deflater
17:38:04.444 INFO FileOutputCommitter - Saved output of task 'attempt_202510271738041667720574155868963_0548_r_000000_0' to file:/tmp/ReadsSparkSinkUnitTest1301386375771717713.bam.parts/_temporary/0/task_202510271738041667720574155868963_0548_r_000000
17:38:04.444 INFO SparkHadoopMapRedUtil - attempt_202510271738041667720574155868963_0548_r_000000_0: Committed. Elapsed time: 0 ms.
17:38:04.445 INFO Executor - Finished task 0.0 in stage 120.0 (TID 176). 1858 bytes result sent to driver
17:38:04.445 INFO TaskSetManager - Finished task 0.0 in stage 120.0 (TID 176) in 47 ms on localhost (executor driver) (1/1)
17:38:04.446 INFO TaskSchedulerImpl - Removed TaskSet 120.0, whose tasks have all completed, from pool
17:38:04.446 INFO DAGScheduler - ResultStage 120 (runJob at SparkHadoopWriter.scala:83) finished in 0.055 s
17:38:04.446 INFO DAGScheduler - Job 83 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:04.446 INFO TaskSchedulerImpl - Killing all running tasks in stage 120: Stage finished
17:38:04.446 INFO DAGScheduler - Job 83 finished: runJob at SparkHadoopWriter.scala:83, took 0.133770 s
17:38:04.446 INFO SparkHadoopWriter - Start to commit write Job job_202510271738041667720574155868963_0548.
17:38:04.451 INFO SparkHadoopWriter - Write Job job_202510271738041667720574155868963_0548 committed. Elapsed time: 4 ms.
17:38:04.454 WARN IntelDeflaterFactory - Intel Deflater not supported, using Java.util.zip.Deflater
17:38:04.460 INFO HadoopFileSystemWrapper - Concatenating 3 parts to /tmp/ReadsSparkSinkUnitTest1301386375771717713.bam
17:38:04.463 INFO HadoopFileSystemWrapper - Concatenating to /tmp/ReadsSparkSinkUnitTest1301386375771717713.bam done
17:38:04.464 INFO IndexFileMerger - Merging .sbi files in temp directory /tmp/ReadsSparkSinkUnitTest1301386375771717713.bam.parts/ to /tmp/ReadsSparkSinkUnitTest1301386375771717713.bam.sbi
17:38:04.469 INFO IndexFileMerger - Done merging .sbi files
17:38:04.469 INFO IndexFileMerger - Merging .bai files in temp directory /tmp/ReadsSparkSinkUnitTest1301386375771717713.bam.parts/ to /tmp/ReadsSparkSinkUnitTest1301386375771717713.bam.bai
17:38:04.473 INFO IndexFileMerger - Done merging .bai files
17:38:04.474 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:04.476 INFO MemoryStore - Block broadcast_221 stored as values in memory (estimated size 320.0 B, free 1918.4 MiB)
17:38:04.476 INFO MemoryStore - Block broadcast_221_piece0 stored as bytes in memory (estimated size 233.0 B, free 1918.4 MiB)
17:38:04.476 INFO BlockManagerInfo - Added broadcast_221_piece0 in memory on localhost:37953 (size: 233.0 B, free: 1919.7 MiB)
17:38:04.477 INFO SparkContext - Created broadcast 221 from broadcast at BamSource.java:104
17:38:04.478 INFO MemoryStore - Block broadcast_222 stored as values in memory (estimated size 297.9 KiB, free 1918.1 MiB)
17:38:04.484 INFO MemoryStore - Block broadcast_222_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1918.0 MiB)
17:38:04.484 INFO BlockManagerInfo - Added broadcast_222_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.7 MiB)
17:38:04.484 INFO SparkContext - Created broadcast 222 from newAPIHadoopFile at PathSplitSource.java:96
17:38:04.494 INFO FileInputFormat - Total input files to process : 1
17:38:04.507 INFO SparkContext - Starting job: collect at ReadsSparkSinkUnitTest.java:182
17:38:04.507 INFO DAGScheduler - Got job 84 (collect at ReadsSparkSinkUnitTest.java:182) with 1 output partitions
17:38:04.507 INFO DAGScheduler - Final stage: ResultStage 121 (collect at ReadsSparkSinkUnitTest.java:182)
17:38:04.507 INFO DAGScheduler - Parents of final stage: List()
17:38:04.508 INFO DAGScheduler - Missing parents: List()
17:38:04.508 INFO DAGScheduler - Submitting ResultStage 121 (MapPartitionsRDD[554] at filter at ReadsSparkSource.java:96), which has no missing parents
17:38:04.513 INFO MemoryStore - Block broadcast_223 stored as values in memory (estimated size 148.2 KiB, free 1917.9 MiB)
17:38:04.514 INFO MemoryStore - Block broadcast_223_piece0 stored as bytes in memory (estimated size 54.6 KiB, free 1917.8 MiB)
17:38:04.514 INFO BlockManagerInfo - Added broadcast_223_piece0 in memory on localhost:37953 (size: 54.6 KiB, free: 1919.6 MiB)
17:38:04.515 INFO SparkContext - Created broadcast 223 from broadcast at DAGScheduler.scala:1580
17:38:04.515 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 121 (MapPartitionsRDD[554] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:38:04.515 INFO TaskSchedulerImpl - Adding task set 121.0 with 1 tasks resource profile 0
17:38:04.515 INFO TaskSetManager - Starting task 0.0 in stage 121.0 (TID 177) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7808 bytes)
17:38:04.516 INFO Executor - Running task 0.0 in stage 121.0 (TID 177)
17:38:04.529 INFO NewHadoopRDD - Input split: file:/tmp/ReadsSparkSinkUnitTest1301386375771717713.bam:0+238969
17:38:04.530 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:04.535 INFO Executor - Finished task 0.0 in stage 121.0 (TID 177). 651483 bytes result sent to driver
17:38:04.536 INFO TaskSetManager - Finished task 0.0 in stage 121.0 (TID 177) in 21 ms on localhost (executor driver) (1/1)
17:38:04.536 INFO TaskSchedulerImpl - Removed TaskSet 121.0, whose tasks have all completed, from pool
17:38:04.537 INFO DAGScheduler - ResultStage 121 (collect at ReadsSparkSinkUnitTest.java:182) finished in 0.028 s
17:38:04.537 INFO DAGScheduler - Job 84 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:04.537 INFO TaskSchedulerImpl - Killing all running tasks in stage 121: Stage finished
17:38:04.537 INFO DAGScheduler - Job 84 finished: collect at ReadsSparkSinkUnitTest.java:182, took 0.029692 s
17:38:04.547 INFO SparkContext - Starting job: count at ReadsSparkSinkUnitTest.java:185
17:38:04.547 INFO DAGScheduler - Got job 85 (count at ReadsSparkSinkUnitTest.java:185) with 1 output partitions
17:38:04.547 INFO DAGScheduler - Final stage: ResultStage 122 (count at ReadsSparkSinkUnitTest.java:185)
17:38:04.547 INFO DAGScheduler - Parents of final stage: List()
17:38:04.547 INFO DAGScheduler - Missing parents: List()
17:38:04.548 INFO DAGScheduler - Submitting ResultStage 122 (MapPartitionsRDD[536] at filter at ReadsSparkSource.java:96), which has no missing parents
17:38:04.563 INFO MemoryStore - Block broadcast_224 stored as values in memory (estimated size 426.1 KiB, free 1917.4 MiB)
17:38:04.564 INFO MemoryStore - Block broadcast_224_piece0 stored as bytes in memory (estimated size 153.6 KiB, free 1917.2 MiB)
17:38:04.565 INFO BlockManagerInfo - Added broadcast_224_piece0 in memory on localhost:37953 (size: 153.6 KiB, free: 1919.5 MiB)
17:38:04.565 INFO SparkContext - Created broadcast 224 from broadcast at DAGScheduler.scala:1580
17:38:04.565 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 122 (MapPartitionsRDD[536] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:38:04.566 INFO TaskSchedulerImpl - Adding task set 122.0 with 1 tasks resource profile 0
17:38:04.566 INFO TaskSetManager - Starting task 0.0 in stage 122.0 (TID 178) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7872 bytes)
17:38:04.567 INFO Executor - Running task 0.0 in stage 122.0 (TID 178)
17:38:04.600 INFO NewHadoopRDD - Input split: file:/home/runner/work/gatk/gatk/src/test/resources/org/broadinstitute/hellbender/tools/BQSR/HiSeq.1mb.1RG.2k_lines.bam:0+222075
17:38:04.601 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:04.602 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:04.610 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:04.614 INFO Executor - Finished task 0.0 in stage 122.0 (TID 178). 989 bytes result sent to driver
17:38:04.614 INFO TaskSetManager - Finished task 0.0 in stage 122.0 (TID 178) in 48 ms on localhost (executor driver) (1/1)
17:38:04.614 INFO TaskSchedulerImpl - Removed TaskSet 122.0, whose tasks have all completed, from pool
17:38:04.614 INFO DAGScheduler - ResultStage 122 (count at ReadsSparkSinkUnitTest.java:185) finished in 0.066 s
17:38:04.615 INFO DAGScheduler - Job 85 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:04.615 INFO TaskSchedulerImpl - Killing all running tasks in stage 122: Stage finished
17:38:04.615 INFO DAGScheduler - Job 85 finished: count at ReadsSparkSinkUnitTest.java:185, took 0.067678 s
17:38:04.618 INFO SparkContext - Starting job: count at ReadsSparkSinkUnitTest.java:185
17:38:04.618 INFO DAGScheduler - Got job 86 (count at ReadsSparkSinkUnitTest.java:185) with 1 output partitions
17:38:04.618 INFO DAGScheduler - Final stage: ResultStage 123 (count at ReadsSparkSinkUnitTest.java:185)
17:38:04.618 INFO DAGScheduler - Parents of final stage: List()
17:38:04.618 INFO DAGScheduler - Missing parents: List()
17:38:04.619 INFO DAGScheduler - Submitting ResultStage 123 (MapPartitionsRDD[554] at filter at ReadsSparkSource.java:96), which has no missing parents
17:38:04.624 INFO MemoryStore - Block broadcast_225 stored as values in memory (estimated size 148.1 KiB, free 1917.1 MiB)
17:38:04.625 INFO MemoryStore - Block broadcast_225_piece0 stored as bytes in memory (estimated size 54.5 KiB, free 1917.1 MiB)
17:38:04.625 INFO BlockManagerInfo - Added broadcast_225_piece0 in memory on localhost:37953 (size: 54.5 KiB, free: 1919.4 MiB)
17:38:04.625 INFO SparkContext - Created broadcast 225 from broadcast at DAGScheduler.scala:1580
17:38:04.625 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 123 (MapPartitionsRDD[554] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:38:04.625 INFO TaskSchedulerImpl - Adding task set 123.0 with 1 tasks resource profile 0
17:38:04.626 INFO TaskSetManager - Starting task 0.0 in stage 123.0 (TID 179) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7808 bytes)
17:38:04.626 INFO Executor - Running task 0.0 in stage 123.0 (TID 179)
17:38:04.639 INFO NewHadoopRDD - Input split: file:/tmp/ReadsSparkSinkUnitTest1301386375771717713.bam:0+238969
17:38:04.640 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:04.644 INFO Executor - Finished task 0.0 in stage 123.0 (TID 179). 989 bytes result sent to driver
17:38:04.644 INFO TaskSetManager - Finished task 0.0 in stage 123.0 (TID 179) in 18 ms on localhost (executor driver) (1/1)
17:38:04.645 INFO TaskSchedulerImpl - Removed TaskSet 123.0, whose tasks have all completed, from pool
17:38:04.645 INFO DAGScheduler - ResultStage 123 (count at ReadsSparkSinkUnitTest.java:185) finished in 0.026 s
17:38:04.645 INFO DAGScheduler - Job 86 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:04.645 INFO TaskSchedulerImpl - Killing all running tasks in stage 123: Stage finished
17:38:04.645 INFO DAGScheduler - Job 86 finished: count at ReadsSparkSinkUnitTest.java:185, took 0.027404 s
17:38:04.646 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:04.649 INFO MemoryStore - Block broadcast_226 stored as values in memory (estimated size 297.9 KiB, free 1916.8 MiB)
17:38:04.655 INFO MemoryStore - Block broadcast_226_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1916.7 MiB)
17:38:04.655 INFO BlockManagerInfo - Added broadcast_226_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.3 MiB)
17:38:04.655 INFO SparkContext - Created broadcast 226 from newAPIHadoopFile at PathSplitSource.java:96
17:38:04.675 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:04.678 INFO MemoryStore - Block broadcast_227 stored as values in memory (estimated size 297.9 KiB, free 1916.4 MiB)
17:38:04.683 INFO BlockManagerInfo - Removed broadcast_218_piece0 on localhost:37953 in memory (size: 9.6 KiB, free: 1919.4 MiB)
17:38:04.684 INFO BlockManagerInfo - Removed broadcast_224_piece0 on localhost:37953 in memory (size: 153.6 KiB, free: 1919.5 MiB)
17:38:04.685 INFO BlockManagerInfo - Removed broadcast_222_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.6 MiB)
17:38:04.685 INFO BlockManagerInfo - Removed broadcast_215_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.6 MiB)
17:38:04.686 INFO BlockManagerInfo - Removed broadcast_223_piece0 on localhost:37953 in memory (size: 54.6 KiB, free: 1919.7 MiB)
17:38:04.686 INFO BlockManagerInfo - Removed broadcast_219_piece0 on localhost:37953 in memory (size: 166.1 KiB, free: 1919.8 MiB)
17:38:04.687 INFO MemoryStore - Block broadcast_227_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1918.7 MiB)
17:38:04.687 INFO BlockManagerInfo - Added broadcast_227_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.8 MiB)
17:38:04.687 INFO SparkContext - Created broadcast 227 from newAPIHadoopFile at PathSplitSource.java:96
17:38:04.687 INFO BlockManagerInfo - Removed broadcast_217_piece0 on localhost:37953 in memory (size: 9.6 KiB, free: 1919.8 MiB)
17:38:04.688 INFO BlockManagerInfo - Removed broadcast_220_piece0 on localhost:37953 in memory (size: 67.0 KiB, free: 1919.8 MiB)
17:38:04.689 INFO BlockManagerInfo - Removed broadcast_221_piece0 on localhost:37953 in memory (size: 233.0 B, free: 1919.8 MiB)
17:38:04.689 INFO BlockManagerInfo - Removed broadcast_225_piece0 on localhost:37953 in memory (size: 54.5 KiB, free: 1919.9 MiB)
17:38:04.708 INFO FileInputFormat - Total input files to process : 1
17:38:04.710 INFO MemoryStore - Block broadcast_228 stored as values in memory (estimated size 160.7 KiB, free 1919.2 MiB)
17:38:04.710 INFO MemoryStore - Block broadcast_228_piece0 stored as bytes in memory (estimated size 9.6 KiB, free 1919.2 MiB)
17:38:04.711 INFO BlockManagerInfo - Added broadcast_228_piece0 in memory on localhost:37953 (size: 9.6 KiB, free: 1919.9 MiB)
17:38:04.711 INFO SparkContext - Created broadcast 228 from broadcast at ReadsSparkSink.java:133
17:38:04.712 INFO MemoryStore - Block broadcast_229 stored as values in memory (estimated size 163.2 KiB, free 1919.0 MiB)
17:38:04.713 INFO MemoryStore - Block broadcast_229_piece0 stored as bytes in memory (estimated size 9.6 KiB, free 1919.0 MiB)
17:38:04.713 INFO BlockManagerInfo - Added broadcast_229_piece0 in memory on localhost:37953 (size: 9.6 KiB, free: 1919.9 MiB)
17:38:04.713 INFO SparkContext - Created broadcast 229 from broadcast at BamSink.java:76
17:38:04.716 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:04.716 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:04.716 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:04.729 INFO SparkContext - Starting job: runJob at SparkHadoopWriter.scala:83
17:38:04.729 INFO DAGScheduler - Registering RDD 568 (mapToPair at SparkUtils.java:161) as input to shuffle 26
17:38:04.730 INFO DAGScheduler - Got job 87 (runJob at SparkHadoopWriter.scala:83) with 1 output partitions
17:38:04.730 INFO DAGScheduler - Final stage: ResultStage 125 (runJob at SparkHadoopWriter.scala:83)
17:38:04.730 INFO DAGScheduler - Parents of final stage: List(ShuffleMapStage 124)
17:38:04.730 INFO DAGScheduler - Missing parents: List(ShuffleMapStage 124)
17:38:04.730 INFO DAGScheduler - Submitting ShuffleMapStage 124 (MapPartitionsRDD[568] at mapToPair at SparkUtils.java:161), which has no missing parents
17:38:04.746 INFO MemoryStore - Block broadcast_230 stored as values in memory (estimated size 520.4 KiB, free 1918.5 MiB)
17:38:04.747 INFO MemoryStore - Block broadcast_230_piece0 stored as bytes in memory (estimated size 166.1 KiB, free 1918.3 MiB)
17:38:04.747 INFO BlockManagerInfo - Added broadcast_230_piece0 in memory on localhost:37953 (size: 166.1 KiB, free: 1919.7 MiB)
17:38:04.748 INFO SparkContext - Created broadcast 230 from broadcast at DAGScheduler.scala:1580
17:38:04.748 INFO DAGScheduler - Submitting 1 missing tasks from ShuffleMapStage 124 (MapPartitionsRDD[568] at mapToPair at SparkUtils.java:161) (first 15 tasks are for partitions Vector(0))
17:38:04.748 INFO TaskSchedulerImpl - Adding task set 124.0 with 1 tasks resource profile 0
17:38:04.749 INFO TaskSetManager - Starting task 0.0 in stage 124.0 (TID 180) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7861 bytes)
17:38:04.749 INFO Executor - Running task 0.0 in stage 124.0 (TID 180)
17:38:04.783 INFO NewHadoopRDD - Input split: file:/home/runner/work/gatk/gatk/src/test/resources/org/broadinstitute/hellbender/tools/BQSR/HiSeq.1mb.1RG.2k_lines.bam:0+222075
17:38:04.784 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:04.785 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:04.792 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:04.801 INFO Executor - Finished task 0.0 in stage 124.0 (TID 180). 1148 bytes result sent to driver
17:38:04.802 INFO TaskSetManager - Finished task 0.0 in stage 124.0 (TID 180) in 54 ms on localhost (executor driver) (1/1)
17:38:04.802 INFO TaskSchedulerImpl - Removed TaskSet 124.0, whose tasks have all completed, from pool
17:38:04.802 INFO DAGScheduler - ShuffleMapStage 124 (mapToPair at SparkUtils.java:161) finished in 0.072 s
17:38:04.802 INFO DAGScheduler - looking for newly runnable stages
17:38:04.802 INFO DAGScheduler - running: HashSet()
17:38:04.802 INFO DAGScheduler - waiting: HashSet(ResultStage 125)
17:38:04.802 INFO DAGScheduler - failed: HashSet()
17:38:04.802 INFO DAGScheduler - Submitting ResultStage 125 (MapPartitionsRDD[573] at mapToPair at BamSink.java:91), which has no missing parents
17:38:04.809 INFO MemoryStore - Block broadcast_231 stored as values in memory (estimated size 241.4 KiB, free 1918.1 MiB)
17:38:04.809 INFO MemoryStore - Block broadcast_231_piece0 stored as bytes in memory (estimated size 67.0 KiB, free 1918.0 MiB)
17:38:04.809 INFO BlockManagerInfo - Added broadcast_231_piece0 in memory on localhost:37953 (size: 67.0 KiB, free: 1919.7 MiB)
17:38:04.810 INFO SparkContext - Created broadcast 231 from broadcast at DAGScheduler.scala:1580
17:38:04.810 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 125 (MapPartitionsRDD[573] at mapToPair at BamSink.java:91) (first 15 tasks are for partitions Vector(0))
17:38:04.810 INFO TaskSchedulerImpl - Adding task set 125.0 with 1 tasks resource profile 0
17:38:04.811 INFO TaskSetManager - Starting task 0.0 in stage 125.0 (TID 181) (localhost, executor driver, partition 0, NODE_LOCAL, 7513 bytes)
17:38:04.811 INFO Executor - Running task 0.0 in stage 125.0 (TID 181)
17:38:04.816 INFO ShuffleBlockFetcherIterator - Getting 1 (343.8 KiB) non-empty blocks including 1 (343.8 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
17:38:04.816 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
17:38:04.829 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:04.829 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:04.829 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:04.829 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:04.829 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:04.829 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:04.837 WARN IntelDeflaterFactory - Intel Deflater not supported, using Java.util.zip.Deflater
17:38:04.854 INFO FileOutputCommitter - Saved output of task 'attempt_202510271738043954116665130056086_0573_r_000000_0' to file:/tmp/ReadsSparkSinkUnitTest13119530199885533548.bam.parts/_temporary/0/task_202510271738043954116665130056086_0573_r_000000
17:38:04.854 INFO SparkHadoopMapRedUtil - attempt_202510271738043954116665130056086_0573_r_000000_0: Committed. Elapsed time: 0 ms.
17:38:04.855 INFO Executor - Finished task 0.0 in stage 125.0 (TID 181). 1858 bytes result sent to driver
17:38:04.855 INFO TaskSetManager - Finished task 0.0 in stage 125.0 (TID 181) in 45 ms on localhost (executor driver) (1/1)
17:38:04.855 INFO TaskSchedulerImpl - Removed TaskSet 125.0, whose tasks have all completed, from pool
17:38:04.856 INFO DAGScheduler - ResultStage 125 (runJob at SparkHadoopWriter.scala:83) finished in 0.053 s
17:38:04.856 INFO DAGScheduler - Job 87 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:04.856 INFO TaskSchedulerImpl - Killing all running tasks in stage 125: Stage finished
17:38:04.856 INFO DAGScheduler - Job 87 finished: runJob at SparkHadoopWriter.scala:83, took 0.126748 s
17:38:04.856 INFO SparkHadoopWriter - Start to commit write Job job_202510271738043954116665130056086_0573.
17:38:04.860 INFO SparkHadoopWriter - Write Job job_202510271738043954116665130056086_0573 committed. Elapsed time: 3 ms.
17:38:04.863 WARN IntelDeflaterFactory - Intel Deflater not supported, using Java.util.zip.Deflater
17:38:04.869 INFO HadoopFileSystemWrapper - Concatenating 3 parts to /tmp/ReadsSparkSinkUnitTest13119530199885533548.bam
17:38:04.872 INFO HadoopFileSystemWrapper - Concatenating to /tmp/ReadsSparkSinkUnitTest13119530199885533548.bam done
17:38:04.872 INFO IndexFileMerger - Merging .sbi files in temp directory /tmp/ReadsSparkSinkUnitTest13119530199885533548.bam.parts/ to /tmp/ReadsSparkSinkUnitTest13119530199885533548.bam.sbi
17:38:04.878 INFO IndexFileMerger - Done merging .sbi files
17:38:04.878 INFO IndexFileMerger - Merging .bai files in temp directory /tmp/ReadsSparkSinkUnitTest13119530199885533548.bam.parts/ to /tmp/ReadsSparkSinkUnitTest13119530199885533548.bam.bai
17:38:04.882 INFO IndexFileMerger - Done merging .bai files
17:38:04.883 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:04.884 INFO MemoryStore - Block broadcast_232 stored as values in memory (estimated size 13.3 KiB, free 1918.0 MiB)
17:38:04.885 INFO MemoryStore - Block broadcast_232_piece0 stored as bytes in memory (estimated size 8.3 KiB, free 1918.0 MiB)
17:38:04.885 INFO BlockManagerInfo - Added broadcast_232_piece0 in memory on localhost:37953 (size: 8.3 KiB, free: 1919.6 MiB)
17:38:04.885 INFO SparkContext - Created broadcast 232 from broadcast at BamSource.java:104
17:38:04.886 INFO MemoryStore - Block broadcast_233 stored as values in memory (estimated size 297.9 KiB, free 1917.7 MiB)
17:38:04.892 INFO MemoryStore - Block broadcast_233_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1917.7 MiB)
17:38:04.892 INFO BlockManagerInfo - Added broadcast_233_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.6 MiB)
17:38:04.892 INFO SparkContext - Created broadcast 233 from newAPIHadoopFile at PathSplitSource.java:96
17:38:04.901 INFO FileInputFormat - Total input files to process : 1
17:38:04.914 INFO SparkContext - Starting job: collect at ReadsSparkSinkUnitTest.java:182
17:38:04.915 INFO DAGScheduler - Got job 88 (collect at ReadsSparkSinkUnitTest.java:182) with 1 output partitions
17:38:04.915 INFO DAGScheduler - Final stage: ResultStage 126 (collect at ReadsSparkSinkUnitTest.java:182)
17:38:04.915 INFO DAGScheduler - Parents of final stage: List()
17:38:04.915 INFO DAGScheduler - Missing parents: List()
17:38:04.915 INFO DAGScheduler - Submitting ResultStage 126 (MapPartitionsRDD[579] at filter at ReadsSparkSource.java:96), which has no missing parents
17:38:04.920 INFO MemoryStore - Block broadcast_234 stored as values in memory (estimated size 148.2 KiB, free 1917.5 MiB)
17:38:04.924 INFO MemoryStore - Block broadcast_234_piece0 stored as bytes in memory (estimated size 54.6 KiB, free 1917.5 MiB)
17:38:04.924 INFO BlockManagerInfo - Removed broadcast_231_piece0 on localhost:37953 in memory (size: 67.0 KiB, free: 1919.7 MiB)
17:38:04.924 INFO BlockManagerInfo - Added broadcast_234_piece0 in memory on localhost:37953 (size: 54.6 KiB, free: 1919.6 MiB)
17:38:04.925 INFO SparkContext - Created broadcast 234 from broadcast at DAGScheduler.scala:1580
17:38:04.925 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 126 (MapPartitionsRDD[579] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:38:04.925 INFO TaskSchedulerImpl - Adding task set 126.0 with 1 tasks resource profile 0
17:38:04.925 INFO BlockManagerInfo - Removed broadcast_229_piece0 on localhost:37953 in memory (size: 9.6 KiB, free: 1919.6 MiB)
17:38:04.925 INFO TaskSetManager - Starting task 0.0 in stage 126.0 (TID 182) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7809 bytes)
17:38:04.925 INFO BlockManagerInfo - Removed broadcast_228_piece0 on localhost:37953 in memory (size: 9.6 KiB, free: 1919.6 MiB)
17:38:04.926 INFO Executor - Running task 0.0 in stage 126.0 (TID 182)
17:38:04.928 INFO BlockManagerInfo - Removed broadcast_230_piece0 on localhost:37953 in memory (size: 166.1 KiB, free: 1919.8 MiB)
17:38:04.928 INFO BlockManagerInfo - Removed broadcast_227_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.8 MiB)
17:38:04.938 INFO NewHadoopRDD - Input split: file:/tmp/ReadsSparkSinkUnitTest13119530199885533548.bam:0+238969
17:38:04.939 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:04.944 INFO Executor - Finished task 0.0 in stage 126.0 (TID 182). 651483 bytes result sent to driver
17:38:04.945 INFO TaskSetManager - Finished task 0.0 in stage 126.0 (TID 182) in 20 ms on localhost (executor driver) (1/1)
17:38:04.945 INFO TaskSchedulerImpl - Removed TaskSet 126.0, whose tasks have all completed, from pool
17:38:04.946 INFO DAGScheduler - ResultStage 126 (collect at ReadsSparkSinkUnitTest.java:182) finished in 0.030 s
17:38:04.946 INFO DAGScheduler - Job 88 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:04.946 INFO TaskSchedulerImpl - Killing all running tasks in stage 126: Stage finished
17:38:04.946 INFO DAGScheduler - Job 88 finished: collect at ReadsSparkSinkUnitTest.java:182, took 0.031405 s
17:38:04.956 INFO SparkContext - Starting job: count at ReadsSparkSinkUnitTest.java:185
17:38:04.956 INFO DAGScheduler - Got job 89 (count at ReadsSparkSinkUnitTest.java:185) with 1 output partitions
17:38:04.956 INFO DAGScheduler - Final stage: ResultStage 127 (count at ReadsSparkSinkUnitTest.java:185)
17:38:04.956 INFO DAGScheduler - Parents of final stage: List()
17:38:04.956 INFO DAGScheduler - Missing parents: List()
17:38:04.956 INFO DAGScheduler - Submitting ResultStage 127 (MapPartitionsRDD[561] at filter at ReadsSparkSource.java:96), which has no missing parents
17:38:04.972 INFO MemoryStore - Block broadcast_235 stored as values in memory (estimated size 426.1 KiB, free 1918.7 MiB)
17:38:04.973 INFO MemoryStore - Block broadcast_235_piece0 stored as bytes in memory (estimated size 153.6 KiB, free 1918.5 MiB)
17:38:04.973 INFO BlockManagerInfo - Added broadcast_235_piece0 in memory on localhost:37953 (size: 153.6 KiB, free: 1919.7 MiB)
17:38:04.973 INFO SparkContext - Created broadcast 235 from broadcast at DAGScheduler.scala:1580
17:38:04.974 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 127 (MapPartitionsRDD[561] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:38:04.974 INFO TaskSchedulerImpl - Adding task set 127.0 with 1 tasks resource profile 0
17:38:04.974 INFO TaskSetManager - Starting task 0.0 in stage 127.0 (TID 183) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7872 bytes)
17:38:04.975 INFO Executor - Running task 0.0 in stage 127.0 (TID 183)
17:38:05.006 INFO NewHadoopRDD - Input split: file:/home/runner/work/gatk/gatk/src/test/resources/org/broadinstitute/hellbender/tools/BQSR/HiSeq.1mb.1RG.2k_lines.bam:0+222075
17:38:05.007 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:05.008 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:05.015 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:05.019 INFO Executor - Finished task 0.0 in stage 127.0 (TID 183). 989 bytes result sent to driver
17:38:05.019 INFO TaskSetManager - Finished task 0.0 in stage 127.0 (TID 183) in 45 ms on localhost (executor driver) (1/1)
17:38:05.019 INFO TaskSchedulerImpl - Removed TaskSet 127.0, whose tasks have all completed, from pool
17:38:05.019 INFO DAGScheduler - ResultStage 127 (count at ReadsSparkSinkUnitTest.java:185) finished in 0.062 s
17:38:05.019 INFO DAGScheduler - Job 89 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:05.020 INFO TaskSchedulerImpl - Killing all running tasks in stage 127: Stage finished
17:38:05.020 INFO DAGScheduler - Job 89 finished: count at ReadsSparkSinkUnitTest.java:185, took 0.063773 s
17:38:05.023 INFO SparkContext - Starting job: count at ReadsSparkSinkUnitTest.java:185
17:38:05.023 INFO DAGScheduler - Got job 90 (count at ReadsSparkSinkUnitTest.java:185) with 1 output partitions
17:38:05.023 INFO DAGScheduler - Final stage: ResultStage 128 (count at ReadsSparkSinkUnitTest.java:185)
17:38:05.023 INFO DAGScheduler - Parents of final stage: List()
17:38:05.023 INFO DAGScheduler - Missing parents: List()
17:38:05.023 INFO DAGScheduler - Submitting ResultStage 128 (MapPartitionsRDD[579] at filter at ReadsSparkSource.java:96), which has no missing parents
17:38:05.029 INFO MemoryStore - Block broadcast_236 stored as values in memory (estimated size 148.1 KiB, free 1918.4 MiB)
17:38:05.029 INFO MemoryStore - Block broadcast_236_piece0 stored as bytes in memory (estimated size 54.5 KiB, free 1918.3 MiB)
17:38:05.030 INFO BlockManagerInfo - Added broadcast_236_piece0 in memory on localhost:37953 (size: 54.5 KiB, free: 1919.6 MiB)
17:38:05.030 INFO SparkContext - Created broadcast 236 from broadcast at DAGScheduler.scala:1580
17:38:05.030 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 128 (MapPartitionsRDD[579] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:38:05.030 INFO TaskSchedulerImpl - Adding task set 128.0 with 1 tasks resource profile 0
17:38:05.031 INFO TaskSetManager - Starting task 0.0 in stage 128.0 (TID 184) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7809 bytes)
17:38:05.031 INFO Executor - Running task 0.0 in stage 128.0 (TID 184)
17:38:05.042 INFO NewHadoopRDD - Input split: file:/tmp/ReadsSparkSinkUnitTest13119530199885533548.bam:0+238969
17:38:05.043 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:05.047 INFO Executor - Finished task 0.0 in stage 128.0 (TID 184). 989 bytes result sent to driver
17:38:05.047 INFO TaskSetManager - Finished task 0.0 in stage 128.0 (TID 184) in 17 ms on localhost (executor driver) (1/1)
17:38:05.047 INFO TaskSchedulerImpl - Removed TaskSet 128.0, whose tasks have all completed, from pool
17:38:05.047 INFO DAGScheduler - ResultStage 128 (count at ReadsSparkSinkUnitTest.java:185) finished in 0.024 s
17:38:05.047 INFO DAGScheduler - Job 90 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:05.047 INFO TaskSchedulerImpl - Killing all running tasks in stage 128: Stage finished
17:38:05.047 INFO DAGScheduler - Job 90 finished: count at ReadsSparkSinkUnitTest.java:185, took 0.024644 s
17:38:05.048 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:05.051 INFO MemoryStore - Block broadcast_237 stored as values in memory (estimated size 297.9 KiB, free 1918.0 MiB)
17:38:05.056 INFO MemoryStore - Block broadcast_237_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1918.0 MiB)
17:38:05.056 INFO BlockManagerInfo - Added broadcast_237_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.6 MiB)
17:38:05.056 INFO SparkContext - Created broadcast 237 from newAPIHadoopFile at PathSplitSource.java:96
17:38:05.075 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:05.077 INFO MemoryStore - Block broadcast_238 stored as values in memory (estimated size 297.9 KiB, free 1917.7 MiB)
17:38:05.083 INFO MemoryStore - Block broadcast_238_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1917.7 MiB)
17:38:05.083 INFO BlockManagerInfo - Added broadcast_238_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.5 MiB)
17:38:05.083 INFO SparkContext - Created broadcast 238 from newAPIHadoopFile at PathSplitSource.java:96
17:38:05.102 INFO FileInputFormat - Total input files to process : 1
17:38:05.104 INFO MemoryStore - Block broadcast_239 stored as values in memory (estimated size 160.7 KiB, free 1917.5 MiB)
17:38:05.105 INFO MemoryStore - Block broadcast_239_piece0 stored as bytes in memory (estimated size 9.6 KiB, free 1917.5 MiB)
17:38:05.105 INFO BlockManagerInfo - Added broadcast_239_piece0 in memory on localhost:37953 (size: 9.6 KiB, free: 1919.5 MiB)
17:38:05.105 INFO SparkContext - Created broadcast 239 from broadcast at ReadsSparkSink.java:133
17:38:05.106 WARN HtsjdkReadsRddStorage - Unrecognized write option: DISABLE
17:38:05.107 INFO MemoryStore - Block broadcast_240 stored as values in memory (estimated size 163.2 KiB, free 1917.3 MiB)
17:38:05.107 INFO MemoryStore - Block broadcast_240_piece0 stored as bytes in memory (estimated size 9.6 KiB, free 1917.3 MiB)
17:38:05.108 INFO BlockManagerInfo - Added broadcast_240_piece0 in memory on localhost:37953 (size: 9.6 KiB, free: 1919.5 MiB)
17:38:05.108 INFO SparkContext - Created broadcast 240 from broadcast at BamSink.java:76
17:38:05.110 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:05.110 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:05.110 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:05.123 INFO SparkContext - Starting job: runJob at SparkHadoopWriter.scala:83
17:38:05.123 INFO DAGScheduler - Registering RDD 593 (mapToPair at SparkUtils.java:161) as input to shuffle 27
17:38:05.123 INFO DAGScheduler - Got job 91 (runJob at SparkHadoopWriter.scala:83) with 1 output partitions
17:38:05.123 INFO DAGScheduler - Final stage: ResultStage 130 (runJob at SparkHadoopWriter.scala:83)
17:38:05.123 INFO DAGScheduler - Parents of final stage: List(ShuffleMapStage 129)
17:38:05.123 INFO DAGScheduler - Missing parents: List(ShuffleMapStage 129)
17:38:05.124 INFO DAGScheduler - Submitting ShuffleMapStage 129 (MapPartitionsRDD[593] at mapToPair at SparkUtils.java:161), which has no missing parents
17:38:05.140 INFO MemoryStore - Block broadcast_241 stored as values in memory (estimated size 520.4 KiB, free 1916.8 MiB)
17:38:05.143 INFO BlockManagerInfo - Removed broadcast_238_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.6 MiB)
17:38:05.144 INFO MemoryStore - Block broadcast_241_piece0 stored as bytes in memory (estimated size 166.1 KiB, free 1917.3 MiB)
17:38:05.144 INFO BlockManagerInfo - Removed broadcast_233_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.6 MiB)
17:38:05.144 INFO BlockManagerInfo - Added broadcast_241_piece0 in memory on localhost:37953 (size: 166.1 KiB, free: 1919.5 MiB)
17:38:05.144 INFO SparkContext - Created broadcast 241 from broadcast at DAGScheduler.scala:1580
17:38:05.144 INFO DAGScheduler - Submitting 1 missing tasks from ShuffleMapStage 129 (MapPartitionsRDD[593] at mapToPair at SparkUtils.java:161) (first 15 tasks are for partitions Vector(0))
17:38:05.144 INFO TaskSchedulerImpl - Adding task set 129.0 with 1 tasks resource profile 0
17:38:05.145 INFO BlockManagerInfo - Removed broadcast_226_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.5 MiB)
17:38:05.145 INFO BlockManagerInfo - Removed broadcast_232_piece0 on localhost:37953 in memory (size: 8.3 KiB, free: 1919.5 MiB)
17:38:05.145 INFO TaskSetManager - Starting task 0.0 in stage 129.0 (TID 185) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7861 bytes)
17:38:05.145 INFO Executor - Running task 0.0 in stage 129.0 (TID 185)
17:38:05.146 INFO BlockManagerInfo - Removed broadcast_235_piece0 on localhost:37953 in memory (size: 153.6 KiB, free: 1919.7 MiB)
17:38:05.146 INFO BlockManagerInfo - Removed broadcast_236_piece0 on localhost:37953 in memory (size: 54.5 KiB, free: 1919.7 MiB)
17:38:05.147 INFO BlockManagerInfo - Removed broadcast_234_piece0 on localhost:37953 in memory (size: 54.6 KiB, free: 1919.8 MiB)
17:38:05.178 INFO NewHadoopRDD - Input split: file:/home/runner/work/gatk/gatk/src/test/resources/org/broadinstitute/hellbender/tools/BQSR/HiSeq.1mb.1RG.2k_lines.bam:0+222075
17:38:05.179 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:05.180 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:05.187 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:05.197 INFO Executor - Finished task 0.0 in stage 129.0 (TID 185). 1148 bytes result sent to driver
17:38:05.197 INFO TaskSetManager - Finished task 0.0 in stage 129.0 (TID 185) in 52 ms on localhost (executor driver) (1/1)
17:38:05.197 INFO TaskSchedulerImpl - Removed TaskSet 129.0, whose tasks have all completed, from pool
17:38:05.197 INFO DAGScheduler - ShuffleMapStage 129 (mapToPair at SparkUtils.java:161) finished in 0.073 s
17:38:05.197 INFO DAGScheduler - looking for newly runnable stages
17:38:05.197 INFO DAGScheduler - running: HashSet()
17:38:05.198 INFO DAGScheduler - waiting: HashSet(ResultStage 130)
17:38:05.198 INFO DAGScheduler - failed: HashSet()
17:38:05.198 INFO DAGScheduler - Submitting ResultStage 130 (MapPartitionsRDD[598] at mapToPair at BamSink.java:91), which has no missing parents
17:38:05.204 INFO MemoryStore - Block broadcast_242 stored as values in memory (estimated size 241.4 KiB, free 1918.4 MiB)
17:38:05.205 INFO MemoryStore - Block broadcast_242_piece0 stored as bytes in memory (estimated size 67.0 KiB, free 1918.4 MiB)
17:38:05.205 INFO BlockManagerInfo - Added broadcast_242_piece0 in memory on localhost:37953 (size: 67.0 KiB, free: 1919.7 MiB)
17:38:05.205 INFO SparkContext - Created broadcast 242 from broadcast at DAGScheduler.scala:1580
17:38:05.206 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 130 (MapPartitionsRDD[598] at mapToPair at BamSink.java:91) (first 15 tasks are for partitions Vector(0))
17:38:05.206 INFO TaskSchedulerImpl - Adding task set 130.0 with 1 tasks resource profile 0
17:38:05.206 INFO TaskSetManager - Starting task 0.0 in stage 130.0 (TID 186) (localhost, executor driver, partition 0, NODE_LOCAL, 7513 bytes)
17:38:05.207 INFO Executor - Running task 0.0 in stage 130.0 (TID 186)
17:38:05.212 INFO ShuffleBlockFetcherIterator - Getting 1 (343.8 KiB) non-empty blocks including 1 (343.8 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
17:38:05.213 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
17:38:05.225 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:05.225 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:05.225 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:05.225 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:05.225 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:05.225 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:05.233 WARN IntelDeflaterFactory - Intel Deflater not supported, using Java.util.zip.Deflater
17:38:05.245 INFO FileOutputCommitter - Saved output of task 'attempt_202510271738056278329182185664869_0598_r_000000_0' to file:/tmp/ReadsSparkSinkUnitTest1183419716039679556.bam.parts/_temporary/0/task_202510271738056278329182185664869_0598_r_000000
17:38:05.245 INFO SparkHadoopMapRedUtil - attempt_202510271738056278329182185664869_0598_r_000000_0: Committed. Elapsed time: 0 ms.
17:38:05.246 INFO Executor - Finished task 0.0 in stage 130.0 (TID 186). 1858 bytes result sent to driver
17:38:05.246 INFO TaskSetManager - Finished task 0.0 in stage 130.0 (TID 186) in 40 ms on localhost (executor driver) (1/1)
17:38:05.246 INFO TaskSchedulerImpl - Removed TaskSet 130.0, whose tasks have all completed, from pool
17:38:05.246 INFO DAGScheduler - ResultStage 130 (runJob at SparkHadoopWriter.scala:83) finished in 0.048 s
17:38:05.246 INFO DAGScheduler - Job 91 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:05.246 INFO TaskSchedulerImpl - Killing all running tasks in stage 130: Stage finished
17:38:05.247 INFO DAGScheduler - Job 91 finished: runJob at SparkHadoopWriter.scala:83, took 0.123890 s
17:38:05.247 INFO SparkHadoopWriter - Start to commit write Job job_202510271738056278329182185664869_0598.
17:38:05.251 INFO SparkHadoopWriter - Write Job job_202510271738056278329182185664869_0598 committed. Elapsed time: 3 ms.
17:38:05.253 WARN IntelDeflaterFactory - Intel Deflater not supported, using Java.util.zip.Deflater
17:38:05.259 INFO HadoopFileSystemWrapper - Concatenating 3 parts to /tmp/ReadsSparkSinkUnitTest1183419716039679556.bam
17:38:05.262 INFO HadoopFileSystemWrapper - Concatenating to /tmp/ReadsSparkSinkUnitTest1183419716039679556.bam done
17:38:05.262 INFO IndexFileMerger - Merging .bai files in temp directory /tmp/ReadsSparkSinkUnitTest1183419716039679556.bam.parts/ to /tmp/ReadsSparkSinkUnitTest1183419716039679556.bam.bai
17:38:05.266 INFO IndexFileMerger - Done merging .bai files
17:38:05.267 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:05.269 INFO MemoryStore - Block broadcast_243 stored as values in memory (estimated size 297.9 KiB, free 1918.1 MiB)
17:38:05.275 INFO MemoryStore - Block broadcast_243_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1918.0 MiB)
17:38:05.275 INFO BlockManagerInfo - Added broadcast_243_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.7 MiB)
17:38:05.275 INFO SparkContext - Created broadcast 243 from newAPIHadoopFile at PathSplitSource.java:96
17:38:05.294 INFO FileInputFormat - Total input files to process : 1
17:38:05.326 INFO SparkContext - Starting job: collect at ReadsSparkSinkUnitTest.java:182
17:38:05.327 INFO DAGScheduler - Got job 92 (collect at ReadsSparkSinkUnitTest.java:182) with 1 output partitions
17:38:05.327 INFO DAGScheduler - Final stage: ResultStage 131 (collect at ReadsSparkSinkUnitTest.java:182)
17:38:05.327 INFO DAGScheduler - Parents of final stage: List()
17:38:05.327 INFO DAGScheduler - Missing parents: List()
17:38:05.327 INFO DAGScheduler - Submitting ResultStage 131 (MapPartitionsRDD[605] at filter at ReadsSparkSource.java:96), which has no missing parents
17:38:05.342 INFO MemoryStore - Block broadcast_244 stored as values in memory (estimated size 426.2 KiB, free 1917.6 MiB)
17:38:05.344 INFO MemoryStore - Block broadcast_244_piece0 stored as bytes in memory (estimated size 153.7 KiB, free 1917.4 MiB)
17:38:05.344 INFO BlockManagerInfo - Added broadcast_244_piece0 in memory on localhost:37953 (size: 153.7 KiB, free: 1919.5 MiB)
17:38:05.344 INFO SparkContext - Created broadcast 244 from broadcast at DAGScheduler.scala:1580
17:38:05.344 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 131 (MapPartitionsRDD[605] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:38:05.344 INFO TaskSchedulerImpl - Adding task set 131.0 with 1 tasks resource profile 0
17:38:05.345 INFO TaskSetManager - Starting task 0.0 in stage 131.0 (TID 187) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7808 bytes)
17:38:05.345 INFO Executor - Running task 0.0 in stage 131.0 (TID 187)
17:38:05.377 INFO NewHadoopRDD - Input split: file:/tmp/ReadsSparkSinkUnitTest1183419716039679556.bam:0+238969
17:38:05.378 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:05.379 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:05.388 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:05.393 INFO Executor - Finished task 0.0 in stage 131.0 (TID 187). 651483 bytes result sent to driver
17:38:05.395 INFO TaskSetManager - Finished task 0.0 in stage 131.0 (TID 187) in 50 ms on localhost (executor driver) (1/1)
17:38:05.395 INFO TaskSchedulerImpl - Removed TaskSet 131.0, whose tasks have all completed, from pool
17:38:05.395 INFO DAGScheduler - ResultStage 131 (collect at ReadsSparkSinkUnitTest.java:182) finished in 0.068 s
17:38:05.395 INFO DAGScheduler - Job 92 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:05.395 INFO TaskSchedulerImpl - Killing all running tasks in stage 131: Stage finished
17:38:05.395 INFO DAGScheduler - Job 92 finished: collect at ReadsSparkSinkUnitTest.java:182, took 0.068401 s
17:38:05.405 INFO SparkContext - Starting job: count at ReadsSparkSinkUnitTest.java:185
17:38:05.405 INFO DAGScheduler - Got job 93 (count at ReadsSparkSinkUnitTest.java:185) with 1 output partitions
17:38:05.405 INFO DAGScheduler - Final stage: ResultStage 132 (count at ReadsSparkSinkUnitTest.java:185)
17:38:05.405 INFO DAGScheduler - Parents of final stage: List()
17:38:05.405 INFO DAGScheduler - Missing parents: List()
17:38:05.405 INFO DAGScheduler - Submitting ResultStage 132 (MapPartitionsRDD[586] at filter at ReadsSparkSource.java:96), which has no missing parents
17:38:05.421 INFO MemoryStore - Block broadcast_245 stored as values in memory (estimated size 426.1 KiB, free 1917.0 MiB)
17:38:05.422 INFO MemoryStore - Block broadcast_245_piece0 stored as bytes in memory (estimated size 153.6 KiB, free 1916.9 MiB)
17:38:05.422 INFO BlockManagerInfo - Added broadcast_245_piece0 in memory on localhost:37953 (size: 153.6 KiB, free: 1919.4 MiB)
17:38:05.422 INFO SparkContext - Created broadcast 245 from broadcast at DAGScheduler.scala:1580
17:38:05.422 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 132 (MapPartitionsRDD[586] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:38:05.422 INFO TaskSchedulerImpl - Adding task set 132.0 with 1 tasks resource profile 0
17:38:05.423 INFO TaskSetManager - Starting task 0.0 in stage 132.0 (TID 188) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7872 bytes)
17:38:05.423 INFO Executor - Running task 0.0 in stage 132.0 (TID 188)
17:38:05.454 INFO NewHadoopRDD - Input split: file:/home/runner/work/gatk/gatk/src/test/resources/org/broadinstitute/hellbender/tools/BQSR/HiSeq.1mb.1RG.2k_lines.bam:0+222075
17:38:05.455 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:05.456 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:05.463 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:05.467 INFO Executor - Finished task 0.0 in stage 132.0 (TID 188). 989 bytes result sent to driver
17:38:05.467 INFO TaskSetManager - Finished task 0.0 in stage 132.0 (TID 188) in 44 ms on localhost (executor driver) (1/1)
17:38:05.467 INFO TaskSchedulerImpl - Removed TaskSet 132.0, whose tasks have all completed, from pool
17:38:05.468 INFO DAGScheduler - ResultStage 132 (count at ReadsSparkSinkUnitTest.java:185) finished in 0.061 s
17:38:05.468 INFO DAGScheduler - Job 93 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:05.468 INFO TaskSchedulerImpl - Killing all running tasks in stage 132: Stage finished
17:38:05.468 INFO DAGScheduler - Job 93 finished: count at ReadsSparkSinkUnitTest.java:185, took 0.062814 s
17:38:05.471 INFO SparkContext - Starting job: count at ReadsSparkSinkUnitTest.java:185
17:38:05.471 INFO DAGScheduler - Got job 94 (count at ReadsSparkSinkUnitTest.java:185) with 1 output partitions
17:38:05.471 INFO DAGScheduler - Final stage: ResultStage 133 (count at ReadsSparkSinkUnitTest.java:185)
17:38:05.471 INFO DAGScheduler - Parents of final stage: List()
17:38:05.471 INFO DAGScheduler - Missing parents: List()
17:38:05.471 INFO DAGScheduler - Submitting ResultStage 133 (MapPartitionsRDD[605] at filter at ReadsSparkSource.java:96), which has no missing parents
17:38:05.486 INFO MemoryStore - Block broadcast_246 stored as values in memory (estimated size 426.1 KiB, free 1916.5 MiB)
17:38:05.487 INFO MemoryStore - Block broadcast_246_piece0 stored as bytes in memory (estimated size 153.6 KiB, free 1916.3 MiB)
17:38:05.488 INFO BlockManagerInfo - Added broadcast_246_piece0 in memory on localhost:37953 (size: 153.6 KiB, free: 1919.2 MiB)
17:38:05.488 INFO SparkContext - Created broadcast 246 from broadcast at DAGScheduler.scala:1580
17:38:05.488 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 133 (MapPartitionsRDD[605] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:38:05.488 INFO TaskSchedulerImpl - Adding task set 133.0 with 1 tasks resource profile 0
17:38:05.489 INFO TaskSetManager - Starting task 0.0 in stage 133.0 (TID 189) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7808 bytes)
17:38:05.489 INFO Executor - Running task 0.0 in stage 133.0 (TID 189)
17:38:05.519 INFO NewHadoopRDD - Input split: file:/tmp/ReadsSparkSinkUnitTest1183419716039679556.bam:0+238969
17:38:05.520 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:05.521 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:05.531 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:05.534 INFO Executor - Finished task 0.0 in stage 133.0 (TID 189). 989 bytes result sent to driver
17:38:05.535 INFO TaskSetManager - Finished task 0.0 in stage 133.0 (TID 189) in 47 ms on localhost (executor driver) (1/1)
17:38:05.535 INFO TaskSchedulerImpl - Removed TaskSet 133.0, whose tasks have all completed, from pool
17:38:05.535 INFO DAGScheduler - ResultStage 133 (count at ReadsSparkSinkUnitTest.java:185) finished in 0.064 s
17:38:05.535 INFO DAGScheduler - Job 94 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:05.535 INFO TaskSchedulerImpl - Killing all running tasks in stage 133: Stage finished
17:38:05.535 INFO DAGScheduler - Job 94 finished: count at ReadsSparkSinkUnitTest.java:185, took 0.064399 s
17:38:05.536 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:05.538 INFO MemoryStore - Block broadcast_247 stored as values in memory (estimated size 297.9 KiB, free 1916.0 MiB)
17:38:05.544 INFO BlockManagerInfo - Removed broadcast_246_piece0 on localhost:37953 in memory (size: 153.6 KiB, free: 1919.4 MiB)
17:38:05.544 INFO BlockManagerInfo - Removed broadcast_245_piece0 on localhost:37953 in memory (size: 153.6 KiB, free: 1919.5 MiB)
17:38:05.545 INFO BlockManagerInfo - Removed broadcast_244_piece0 on localhost:37953 in memory (size: 153.7 KiB, free: 1919.7 MiB)
17:38:05.545 INFO BlockManagerInfo - Removed broadcast_237_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.7 MiB)
17:38:05.546 INFO BlockManagerInfo - Removed broadcast_243_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.8 MiB)
17:38:05.546 INFO BlockManagerInfo - Removed broadcast_240_piece0 on localhost:37953 in memory (size: 9.6 KiB, free: 1919.8 MiB)
17:38:05.547 INFO BlockManagerInfo - Removed broadcast_242_piece0 on localhost:37953 in memory (size: 67.0 KiB, free: 1919.8 MiB)
17:38:05.547 INFO MemoryStore - Block broadcast_247_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1918.8 MiB)
17:38:05.547 INFO BlockManagerInfo - Added broadcast_247_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.8 MiB)
17:38:05.547 INFO SparkContext - Created broadcast 247 from newAPIHadoopFile at PathSplitSource.java:96
17:38:05.548 INFO BlockManagerInfo - Removed broadcast_241_piece0 on localhost:37953 in memory (size: 166.1 KiB, free: 1919.9 MiB)
17:38:05.549 INFO BlockManagerInfo - Removed broadcast_239_piece0 on localhost:37953 in memory (size: 9.6 KiB, free: 1920.0 MiB)
17:38:05.566 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:05.568 INFO MemoryStore - Block broadcast_248 stored as values in memory (estimated size 297.9 KiB, free 1919.4 MiB)
17:38:05.574 INFO MemoryStore - Block broadcast_248_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1919.3 MiB)
17:38:05.574 INFO BlockManagerInfo - Added broadcast_248_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.9 MiB)
17:38:05.574 INFO SparkContext - Created broadcast 248 from newAPIHadoopFile at PathSplitSource.java:96
17:38:05.593 INFO FileInputFormat - Total input files to process : 1
17:38:05.595 INFO MemoryStore - Block broadcast_249 stored as values in memory (estimated size 160.7 KiB, free 1919.2 MiB)
17:38:05.595 INFO MemoryStore - Block broadcast_249_piece0 stored as bytes in memory (estimated size 9.6 KiB, free 1919.2 MiB)
17:38:05.596 INFO BlockManagerInfo - Added broadcast_249_piece0 in memory on localhost:37953 (size: 9.6 KiB, free: 1919.9 MiB)
17:38:05.596 INFO SparkContext - Created broadcast 249 from broadcast at ReadsSparkSink.java:133
17:38:05.596 WARN HtsjdkReadsRddStorage - Unrecognized write option: DISABLE
17:38:05.597 INFO MemoryStore - Block broadcast_250 stored as values in memory (estimated size 163.2 KiB, free 1919.0 MiB)
17:38:05.598 INFO MemoryStore - Block broadcast_250_piece0 stored as bytes in memory (estimated size 9.6 KiB, free 1919.0 MiB)
17:38:05.598 INFO BlockManagerInfo - Added broadcast_250_piece0 in memory on localhost:37953 (size: 9.6 KiB, free: 1919.9 MiB)
17:38:05.598 INFO SparkContext - Created broadcast 250 from broadcast at BamSink.java:76
17:38:05.600 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:05.600 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:05.600 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:05.613 INFO SparkContext - Starting job: runJob at SparkHadoopWriter.scala:83
17:38:05.614 INFO DAGScheduler - Registering RDD 619 (mapToPair at SparkUtils.java:161) as input to shuffle 28
17:38:05.614 INFO DAGScheduler - Got job 95 (runJob at SparkHadoopWriter.scala:83) with 1 output partitions
17:38:05.614 INFO DAGScheduler - Final stage: ResultStage 135 (runJob at SparkHadoopWriter.scala:83)
17:38:05.614 INFO DAGScheduler - Parents of final stage: List(ShuffleMapStage 134)
17:38:05.614 INFO DAGScheduler - Missing parents: List(ShuffleMapStage 134)
17:38:05.614 INFO DAGScheduler - Submitting ShuffleMapStage 134 (MapPartitionsRDD[619] at mapToPair at SparkUtils.java:161), which has no missing parents
17:38:05.630 INFO MemoryStore - Block broadcast_251 stored as values in memory (estimated size 520.4 KiB, free 1918.5 MiB)
17:38:05.631 INFO MemoryStore - Block broadcast_251_piece0 stored as bytes in memory (estimated size 166.1 KiB, free 1918.3 MiB)
17:38:05.631 INFO BlockManagerInfo - Added broadcast_251_piece0 in memory on localhost:37953 (size: 166.1 KiB, free: 1919.7 MiB)
17:38:05.631 INFO SparkContext - Created broadcast 251 from broadcast at DAGScheduler.scala:1580
17:38:05.631 INFO DAGScheduler - Submitting 1 missing tasks from ShuffleMapStage 134 (MapPartitionsRDD[619] at mapToPair at SparkUtils.java:161) (first 15 tasks are for partitions Vector(0))
17:38:05.631 INFO TaskSchedulerImpl - Adding task set 134.0 with 1 tasks resource profile 0
17:38:05.632 INFO TaskSetManager - Starting task 0.0 in stage 134.0 (TID 190) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7861 bytes)
17:38:05.632 INFO Executor - Running task 0.0 in stage 134.0 (TID 190)
17:38:05.664 INFO NewHadoopRDD - Input split: file:/home/runner/work/gatk/gatk/src/test/resources/org/broadinstitute/hellbender/tools/BQSR/HiSeq.1mb.1RG.2k_lines.bam:0+222075
17:38:05.665 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:05.666 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:05.673 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:05.682 INFO Executor - Finished task 0.0 in stage 134.0 (TID 190). 1148 bytes result sent to driver
17:38:05.682 INFO TaskSetManager - Finished task 0.0 in stage 134.0 (TID 190) in 50 ms on localhost (executor driver) (1/1)
17:38:05.682 INFO TaskSchedulerImpl - Removed TaskSet 134.0, whose tasks have all completed, from pool
17:38:05.682 INFO DAGScheduler - ShuffleMapStage 134 (mapToPair at SparkUtils.java:161) finished in 0.068 s
17:38:05.682 INFO DAGScheduler - looking for newly runnable stages
17:38:05.683 INFO DAGScheduler - running: HashSet()
17:38:05.683 INFO DAGScheduler - waiting: HashSet(ResultStage 135)
17:38:05.683 INFO DAGScheduler - failed: HashSet()
17:38:05.683 INFO DAGScheduler - Submitting ResultStage 135 (MapPartitionsRDD[624] at mapToPair at BamSink.java:91), which has no missing parents
17:38:05.689 INFO MemoryStore - Block broadcast_252 stored as values in memory (estimated size 241.4 KiB, free 1918.1 MiB)
17:38:05.689 INFO MemoryStore - Block broadcast_252_piece0 stored as bytes in memory (estimated size 67.0 KiB, free 1918.0 MiB)
17:38:05.690 INFO BlockManagerInfo - Added broadcast_252_piece0 in memory on localhost:37953 (size: 67.0 KiB, free: 1919.7 MiB)
17:38:05.690 INFO SparkContext - Created broadcast 252 from broadcast at DAGScheduler.scala:1580
17:38:05.690 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 135 (MapPartitionsRDD[624] at mapToPair at BamSink.java:91) (first 15 tasks are for partitions Vector(0))
17:38:05.690 INFO TaskSchedulerImpl - Adding task set 135.0 with 1 tasks resource profile 0
17:38:05.690 INFO TaskSetManager - Starting task 0.0 in stage 135.0 (TID 191) (localhost, executor driver, partition 0, NODE_LOCAL, 7513 bytes)
17:38:05.691 INFO Executor - Running task 0.0 in stage 135.0 (TID 191)
17:38:05.696 INFO ShuffleBlockFetcherIterator - Getting 1 (343.8 KiB) non-empty blocks including 1 (343.8 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
17:38:05.696 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
17:38:05.707 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:05.707 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:05.707 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:05.707 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:05.707 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:05.707 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:05.715 WARN IntelDeflaterFactory - Intel Deflater not supported, using Java.util.zip.Deflater
17:38:05.727 INFO FileOutputCommitter - Saved output of task 'attempt_202510271738051093660346673672503_0624_r_000000_0' to file:/tmp/ReadsSparkSinkUnitTest15940144163841202201.bam.parts/_temporary/0/task_202510271738051093660346673672503_0624_r_000000
17:38:05.727 INFO SparkHadoopMapRedUtil - attempt_202510271738051093660346673672503_0624_r_000000_0: Committed. Elapsed time: 0 ms.
17:38:05.728 INFO Executor - Finished task 0.0 in stage 135.0 (TID 191). 1858 bytes result sent to driver
17:38:05.729 INFO TaskSetManager - Finished task 0.0 in stage 135.0 (TID 191) in 39 ms on localhost (executor driver) (1/1)
17:38:05.729 INFO TaskSchedulerImpl - Removed TaskSet 135.0, whose tasks have all completed, from pool
17:38:05.729 INFO DAGScheduler - ResultStage 135 (runJob at SparkHadoopWriter.scala:83) finished in 0.046 s
17:38:05.729 INFO DAGScheduler - Job 95 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:05.729 INFO TaskSchedulerImpl - Killing all running tasks in stage 135: Stage finished
17:38:05.729 INFO DAGScheduler - Job 95 finished: runJob at SparkHadoopWriter.scala:83, took 0.115890 s
17:38:05.729 INFO SparkHadoopWriter - Start to commit write Job job_202510271738051093660346673672503_0624.
17:38:05.733 INFO SparkHadoopWriter - Write Job job_202510271738051093660346673672503_0624 committed. Elapsed time: 3 ms.
17:38:05.735 WARN IntelDeflaterFactory - Intel Deflater not supported, using Java.util.zip.Deflater
17:38:05.741 INFO HadoopFileSystemWrapper - Concatenating 3 parts to /tmp/ReadsSparkSinkUnitTest15940144163841202201.bam
17:38:05.744 INFO HadoopFileSystemWrapper - Concatenating to /tmp/ReadsSparkSinkUnitTest15940144163841202201.bam done
17:38:05.745 INFO IndexFileMerger - Merging .sbi files in temp directory /tmp/ReadsSparkSinkUnitTest15940144163841202201.bam.parts/ to /tmp/ReadsSparkSinkUnitTest15940144163841202201.bam.sbi
17:38:05.749 INFO IndexFileMerger - Done merging .sbi files
17:38:05.750 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:05.751 INFO MemoryStore - Block broadcast_253 stored as values in memory (estimated size 320.0 B, free 1918.0 MiB)
17:38:05.752 INFO MemoryStore - Block broadcast_253_piece0 stored as bytes in memory (estimated size 233.0 B, free 1918.0 MiB)
17:38:05.752 INFO BlockManagerInfo - Added broadcast_253_piece0 in memory on localhost:37953 (size: 233.0 B, free: 1919.7 MiB)
17:38:05.752 INFO SparkContext - Created broadcast 253 from broadcast at BamSource.java:104
17:38:05.753 INFO MemoryStore - Block broadcast_254 stored as values in memory (estimated size 297.9 KiB, free 1917.7 MiB)
17:38:05.758 INFO BlockManagerInfo - Removed broadcast_252_piece0 on localhost:37953 in memory (size: 67.0 KiB, free: 1919.7 MiB)
17:38:05.759 INFO BlockManagerInfo - Removed broadcast_250_piece0 on localhost:37953 in memory (size: 9.6 KiB, free: 1919.7 MiB)
17:38:05.760 INFO BlockManagerInfo - Removed broadcast_251_piece0 on localhost:37953 in memory (size: 166.1 KiB, free: 1919.9 MiB)
17:38:05.760 INFO BlockManagerInfo - Removed broadcast_249_piece0 on localhost:37953 in memory (size: 9.6 KiB, free: 1919.9 MiB)
17:38:05.761 INFO BlockManagerInfo - Removed broadcast_248_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1920.0 MiB)
17:38:05.762 INFO MemoryStore - Block broadcast_254_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1919.3 MiB)
17:38:05.762 INFO BlockManagerInfo - Added broadcast_254_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.9 MiB)
17:38:05.762 INFO SparkContext - Created broadcast 254 from newAPIHadoopFile at PathSplitSource.java:96
17:38:05.771 INFO FileInputFormat - Total input files to process : 1
17:38:05.785 INFO SparkContext - Starting job: collect at ReadsSparkSinkUnitTest.java:182
17:38:05.785 INFO DAGScheduler - Got job 96 (collect at ReadsSparkSinkUnitTest.java:182) with 1 output partitions
17:38:05.785 INFO DAGScheduler - Final stage: ResultStage 136 (collect at ReadsSparkSinkUnitTest.java:182)
17:38:05.785 INFO DAGScheduler - Parents of final stage: List()
17:38:05.785 INFO DAGScheduler - Missing parents: List()
17:38:05.785 INFO DAGScheduler - Submitting ResultStage 136 (MapPartitionsRDD[630] at filter at ReadsSparkSource.java:96), which has no missing parents
17:38:05.791 INFO MemoryStore - Block broadcast_255 stored as values in memory (estimated size 148.2 KiB, free 1919.2 MiB)
17:38:05.792 INFO MemoryStore - Block broadcast_255_piece0 stored as bytes in memory (estimated size 54.6 KiB, free 1919.1 MiB)
17:38:05.792 INFO BlockManagerInfo - Added broadcast_255_piece0 in memory on localhost:37953 (size: 54.6 KiB, free: 1919.8 MiB)
17:38:05.792 INFO SparkContext - Created broadcast 255 from broadcast at DAGScheduler.scala:1580
17:38:05.792 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 136 (MapPartitionsRDD[630] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:38:05.792 INFO TaskSchedulerImpl - Adding task set 136.0 with 1 tasks resource profile 0
17:38:05.793 INFO TaskSetManager - Starting task 0.0 in stage 136.0 (TID 192) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7809 bytes)
17:38:05.794 INFO Executor - Running task 0.0 in stage 136.0 (TID 192)
17:38:05.806 INFO NewHadoopRDD - Input split: file:/tmp/ReadsSparkSinkUnitTest15940144163841202201.bam:0+238969
17:38:05.807 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:05.812 INFO Executor - Finished task 0.0 in stage 136.0 (TID 192). 651483 bytes result sent to driver
17:38:05.813 INFO TaskSetManager - Finished task 0.0 in stage 136.0 (TID 192) in 20 ms on localhost (executor driver) (1/1)
17:38:05.813 INFO TaskSchedulerImpl - Removed TaskSet 136.0, whose tasks have all completed, from pool
17:38:05.813 INFO DAGScheduler - ResultStage 136 (collect at ReadsSparkSinkUnitTest.java:182) finished in 0.027 s
17:38:05.813 INFO DAGScheduler - Job 96 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:05.813 INFO TaskSchedulerImpl - Killing all running tasks in stage 136: Stage finished
17:38:05.814 INFO DAGScheduler - Job 96 finished: collect at ReadsSparkSinkUnitTest.java:182, took 0.028667 s
17:38:05.823 INFO SparkContext - Starting job: count at ReadsSparkSinkUnitTest.java:185
17:38:05.824 INFO DAGScheduler - Got job 97 (count at ReadsSparkSinkUnitTest.java:185) with 1 output partitions
17:38:05.824 INFO DAGScheduler - Final stage: ResultStage 137 (count at ReadsSparkSinkUnitTest.java:185)
17:38:05.824 INFO DAGScheduler - Parents of final stage: List()
17:38:05.824 INFO DAGScheduler - Missing parents: List()
17:38:05.824 INFO DAGScheduler - Submitting ResultStage 137 (MapPartitionsRDD[612] at filter at ReadsSparkSource.java:96), which has no missing parents
17:38:05.839 INFO MemoryStore - Block broadcast_256 stored as values in memory (estimated size 426.1 KiB, free 1918.7 MiB)
17:38:05.840 INFO MemoryStore - Block broadcast_256_piece0 stored as bytes in memory (estimated size 153.6 KiB, free 1918.6 MiB)
17:38:05.840 INFO BlockManagerInfo - Added broadcast_256_piece0 in memory on localhost:37953 (size: 153.6 KiB, free: 1919.7 MiB)
17:38:05.841 INFO SparkContext - Created broadcast 256 from broadcast at DAGScheduler.scala:1580
17:38:05.841 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 137 (MapPartitionsRDD[612] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:38:05.841 INFO TaskSchedulerImpl - Adding task set 137.0 with 1 tasks resource profile 0
17:38:05.842 INFO TaskSetManager - Starting task 0.0 in stage 137.0 (TID 193) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7872 bytes)
17:38:05.842 INFO Executor - Running task 0.0 in stage 137.0 (TID 193)
17:38:05.873 INFO NewHadoopRDD - Input split: file:/home/runner/work/gatk/gatk/src/test/resources/org/broadinstitute/hellbender/tools/BQSR/HiSeq.1mb.1RG.2k_lines.bam:0+222075
17:38:05.874 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:05.875 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:05.882 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:05.885 INFO Executor - Finished task 0.0 in stage 137.0 (TID 193). 989 bytes result sent to driver
17:38:05.886 INFO TaskSetManager - Finished task 0.0 in stage 137.0 (TID 193) in 45 ms on localhost (executor driver) (1/1)
17:38:05.886 INFO TaskSchedulerImpl - Removed TaskSet 137.0, whose tasks have all completed, from pool
17:38:05.886 INFO DAGScheduler - ResultStage 137 (count at ReadsSparkSinkUnitTest.java:185) finished in 0.062 s
17:38:05.886 INFO DAGScheduler - Job 97 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:05.886 INFO TaskSchedulerImpl - Killing all running tasks in stage 137: Stage finished
17:38:05.886 INFO DAGScheduler - Job 97 finished: count at ReadsSparkSinkUnitTest.java:185, took 0.062725 s
17:38:05.889 INFO SparkContext - Starting job: count at ReadsSparkSinkUnitTest.java:185
17:38:05.890 INFO DAGScheduler - Got job 98 (count at ReadsSparkSinkUnitTest.java:185) with 1 output partitions
17:38:05.890 INFO DAGScheduler - Final stage: ResultStage 138 (count at ReadsSparkSinkUnitTest.java:185)
17:38:05.890 INFO DAGScheduler - Parents of final stage: List()
17:38:05.890 INFO DAGScheduler - Missing parents: List()
17:38:05.890 INFO DAGScheduler - Submitting ResultStage 138 (MapPartitionsRDD[630] at filter at ReadsSparkSource.java:96), which has no missing parents
17:38:05.895 INFO MemoryStore - Block broadcast_257 stored as values in memory (estimated size 148.1 KiB, free 1918.4 MiB)
17:38:05.896 INFO MemoryStore - Block broadcast_257_piece0 stored as bytes in memory (estimated size 54.5 KiB, free 1918.4 MiB)
17:38:05.896 INFO BlockManagerInfo - Added broadcast_257_piece0 in memory on localhost:37953 (size: 54.5 KiB, free: 1919.6 MiB)
17:38:05.896 INFO SparkContext - Created broadcast 257 from broadcast at DAGScheduler.scala:1580
17:38:05.896 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 138 (MapPartitionsRDD[630] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:38:05.896 INFO TaskSchedulerImpl - Adding task set 138.0 with 1 tasks resource profile 0
17:38:05.897 INFO TaskSetManager - Starting task 0.0 in stage 138.0 (TID 194) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7809 bytes)
17:38:05.897 INFO Executor - Running task 0.0 in stage 138.0 (TID 194)
17:38:05.908 INFO NewHadoopRDD - Input split: file:/tmp/ReadsSparkSinkUnitTest15940144163841202201.bam:0+238969
17:38:05.908 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:05.912 INFO Executor - Finished task 0.0 in stage 138.0 (TID 194). 989 bytes result sent to driver
17:38:05.912 INFO TaskSetManager - Finished task 0.0 in stage 138.0 (TID 194) in 15 ms on localhost (executor driver) (1/1)
17:38:05.912 INFO TaskSchedulerImpl - Removed TaskSet 138.0, whose tasks have all completed, from pool
17:38:05.912 INFO DAGScheduler - ResultStage 138 (count at ReadsSparkSinkUnitTest.java:185) finished in 0.022 s
17:38:05.913 INFO DAGScheduler - Job 98 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:05.913 INFO TaskSchedulerImpl - Killing all running tasks in stage 138: Stage finished
17:38:05.913 INFO DAGScheduler - Job 98 finished: count at ReadsSparkSinkUnitTest.java:185, took 0.023302 s
17:38:05.914 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:05.916 INFO MemoryStore - Block broadcast_258 stored as values in memory (estimated size 297.9 KiB, free 1918.1 MiB)
17:38:05.921 INFO MemoryStore - Block broadcast_258_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1918.0 MiB)
17:38:05.921 INFO BlockManagerInfo - Added broadcast_258_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.6 MiB)
17:38:05.921 INFO SparkContext - Created broadcast 258 from newAPIHadoopFile at PathSplitSource.java:96
17:38:05.940 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:05.942 INFO MemoryStore - Block broadcast_259 stored as values in memory (estimated size 297.9 KiB, free 1917.7 MiB)
17:38:05.948 INFO MemoryStore - Block broadcast_259_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1917.7 MiB)
17:38:05.948 INFO BlockManagerInfo - Added broadcast_259_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.5 MiB)
17:38:05.948 INFO SparkContext - Created broadcast 259 from newAPIHadoopFile at PathSplitSource.java:96
17:38:05.967 INFO FileInputFormat - Total input files to process : 1
17:38:05.969 INFO MemoryStore - Block broadcast_260 stored as values in memory (estimated size 160.7 KiB, free 1917.5 MiB)
17:38:05.969 INFO MemoryStore - Block broadcast_260_piece0 stored as bytes in memory (estimated size 9.6 KiB, free 1917.5 MiB)
17:38:05.970 INFO BlockManagerInfo - Added broadcast_260_piece0 in memory on localhost:37953 (size: 9.6 KiB, free: 1919.5 MiB)
17:38:05.970 INFO SparkContext - Created broadcast 260 from broadcast at ReadsSparkSink.java:133
17:38:05.970 WARN HtsjdkReadsRddStorage - Unrecognized write option: DISABLE
17:38:05.970 WARN HtsjdkReadsRddStorage - Unrecognized write option: DISABLE
17:38:05.971 INFO MemoryStore - Block broadcast_261 stored as values in memory (estimated size 163.2 KiB, free 1917.4 MiB)
17:38:05.975 INFO MemoryStore - Block broadcast_261_piece0 stored as bytes in memory (estimated size 9.6 KiB, free 1917.5 MiB)
17:38:05.975 INFO BlockManagerInfo - Removed broadcast_257_piece0 on localhost:37953 in memory (size: 54.5 KiB, free: 1919.6 MiB)
17:38:05.975 INFO BlockManagerInfo - Added broadcast_261_piece0 in memory on localhost:37953 (size: 9.6 KiB, free: 1919.6 MiB)
17:38:05.975 INFO BlockManagerInfo - Removed broadcast_256_piece0 on localhost:37953 in memory (size: 153.6 KiB, free: 1919.7 MiB)
17:38:05.976 INFO SparkContext - Created broadcast 261 from broadcast at BamSink.java:76
17:38:05.976 INFO BlockManagerInfo - Removed broadcast_247_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.8 MiB)
17:38:05.976 INFO BlockManagerInfo - Removed broadcast_255_piece0 on localhost:37953 in memory (size: 54.6 KiB, free: 1919.8 MiB)
17:38:05.977 INFO BlockManagerInfo - Removed broadcast_253_piece0 on localhost:37953 in memory (size: 233.0 B, free: 1919.8 MiB)
17:38:05.978 INFO BlockManagerInfo - Removed broadcast_259_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.9 MiB)
17:38:05.978 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:05.978 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:05.978 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:05.978 INFO BlockManagerInfo - Removed broadcast_254_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.9 MiB)
17:38:05.990 INFO SparkContext - Starting job: runJob at SparkHadoopWriter.scala:83
17:38:05.991 INFO DAGScheduler - Registering RDD 644 (mapToPair at SparkUtils.java:161) as input to shuffle 29
17:38:05.991 INFO DAGScheduler - Got job 99 (runJob at SparkHadoopWriter.scala:83) with 1 output partitions
17:38:05.991 INFO DAGScheduler - Final stage: ResultStage 140 (runJob at SparkHadoopWriter.scala:83)
17:38:05.991 INFO DAGScheduler - Parents of final stage: List(ShuffleMapStage 139)
17:38:05.991 INFO DAGScheduler - Missing parents: List(ShuffleMapStage 139)
17:38:05.991 INFO DAGScheduler - Submitting ShuffleMapStage 139 (MapPartitionsRDD[644] at mapToPair at SparkUtils.java:161), which has no missing parents
17:38:06.007 INFO MemoryStore - Block broadcast_262 stored as values in memory (estimated size 520.4 KiB, free 1918.8 MiB)
17:38:06.008 INFO MemoryStore - Block broadcast_262_piece0 stored as bytes in memory (estimated size 166.1 KiB, free 1918.7 MiB)
17:38:06.009 INFO BlockManagerInfo - Added broadcast_262_piece0 in memory on localhost:37953 (size: 166.1 KiB, free: 1919.8 MiB)
17:38:06.009 INFO SparkContext - Created broadcast 262 from broadcast at DAGScheduler.scala:1580
17:38:06.009 INFO DAGScheduler - Submitting 1 missing tasks from ShuffleMapStage 139 (MapPartitionsRDD[644] at mapToPair at SparkUtils.java:161) (first 15 tasks are for partitions Vector(0))
17:38:06.009 INFO TaskSchedulerImpl - Adding task set 139.0 with 1 tasks resource profile 0
17:38:06.010 INFO TaskSetManager - Starting task 0.0 in stage 139.0 (TID 195) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7861 bytes)
17:38:06.010 INFO Executor - Running task 0.0 in stage 139.0 (TID 195)
17:38:06.042 INFO NewHadoopRDD - Input split: file:/home/runner/work/gatk/gatk/src/test/resources/org/broadinstitute/hellbender/tools/BQSR/HiSeq.1mb.1RG.2k_lines.bam:0+222075
17:38:06.043 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:06.044 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:06.051 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:06.060 INFO Executor - Finished task 0.0 in stage 139.0 (TID 195). 1148 bytes result sent to driver
17:38:06.061 INFO TaskSetManager - Finished task 0.0 in stage 139.0 (TID 195) in 51 ms on localhost (executor driver) (1/1)
17:38:06.061 INFO TaskSchedulerImpl - Removed TaskSet 139.0, whose tasks have all completed, from pool
17:38:06.061 INFO DAGScheduler - ShuffleMapStage 139 (mapToPair at SparkUtils.java:161) finished in 0.069 s
17:38:06.061 INFO DAGScheduler - looking for newly runnable stages
17:38:06.061 INFO DAGScheduler - running: HashSet()
17:38:06.061 INFO DAGScheduler - waiting: HashSet(ResultStage 140)
17:38:06.061 INFO DAGScheduler - failed: HashSet()
17:38:06.061 INFO DAGScheduler - Submitting ResultStage 140 (MapPartitionsRDD[649] at mapToPair at BamSink.java:91), which has no missing parents
17:38:06.067 INFO MemoryStore - Block broadcast_263 stored as values in memory (estimated size 241.4 KiB, free 1918.4 MiB)
17:38:06.068 INFO MemoryStore - Block broadcast_263_piece0 stored as bytes in memory (estimated size 67.0 KiB, free 1918.4 MiB)
17:38:06.068 INFO BlockManagerInfo - Added broadcast_263_piece0 in memory on localhost:37953 (size: 67.0 KiB, free: 1919.7 MiB)
17:38:06.068 INFO SparkContext - Created broadcast 263 from broadcast at DAGScheduler.scala:1580
17:38:06.068 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 140 (MapPartitionsRDD[649] at mapToPair at BamSink.java:91) (first 15 tasks are for partitions Vector(0))
17:38:06.068 INFO TaskSchedulerImpl - Adding task set 140.0 with 1 tasks resource profile 0
17:38:06.069 INFO TaskSetManager - Starting task 0.0 in stage 140.0 (TID 196) (localhost, executor driver, partition 0, NODE_LOCAL, 7513 bytes)
17:38:06.069 INFO Executor - Running task 0.0 in stage 140.0 (TID 196)
17:38:06.074 INFO ShuffleBlockFetcherIterator - Getting 1 (343.8 KiB) non-empty blocks including 1 (343.8 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
17:38:06.074 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
17:38:06.085 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:06.086 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:06.086 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:06.086 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:06.086 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:06.086 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:06.093 WARN IntelDeflaterFactory - Intel Deflater not supported, using Java.util.zip.Deflater
17:38:06.102 INFO FileOutputCommitter - Saved output of task 'attempt_202510271738054630722824340283501_0649_r_000000_0' to file:/tmp/ReadsSparkSinkUnitTest19999427906069391149.bam.parts/_temporary/0/task_202510271738054630722824340283501_0649_r_000000
17:38:06.102 INFO SparkHadoopMapRedUtil - attempt_202510271738054630722824340283501_0649_r_000000_0: Committed. Elapsed time: 0 ms.
17:38:06.102 INFO Executor - Finished task 0.0 in stage 140.0 (TID 196). 1858 bytes result sent to driver
17:38:06.102 INFO TaskSetManager - Finished task 0.0 in stage 140.0 (TID 196) in 33 ms on localhost (executor driver) (1/1)
17:38:06.103 INFO TaskSchedulerImpl - Removed TaskSet 140.0, whose tasks have all completed, from pool
17:38:06.103 INFO DAGScheduler - ResultStage 140 (runJob at SparkHadoopWriter.scala:83) finished in 0.042 s
17:38:06.103 INFO DAGScheduler - Job 99 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:06.103 INFO TaskSchedulerImpl - Killing all running tasks in stage 140: Stage finished
17:38:06.103 INFO DAGScheduler - Job 99 finished: runJob at SparkHadoopWriter.scala:83, took 0.112357 s
17:38:06.103 INFO SparkHadoopWriter - Start to commit write Job job_202510271738054630722824340283501_0649.
17:38:06.106 INFO SparkHadoopWriter - Write Job job_202510271738054630722824340283501_0649 committed. Elapsed time: 3 ms.
17:38:06.109 WARN IntelDeflaterFactory - Intel Deflater not supported, using Java.util.zip.Deflater
17:38:06.114 INFO HadoopFileSystemWrapper - Concatenating 3 parts to /tmp/ReadsSparkSinkUnitTest19999427906069391149.bam
17:38:06.117 INFO HadoopFileSystemWrapper - Concatenating to /tmp/ReadsSparkSinkUnitTest19999427906069391149.bam done
17:38:06.118 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:06.120 INFO MemoryStore - Block broadcast_264 stored as values in memory (estimated size 297.9 KiB, free 1918.1 MiB)
17:38:06.125 INFO MemoryStore - Block broadcast_264_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1918.0 MiB)
17:38:06.125 INFO BlockManagerInfo - Added broadcast_264_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.7 MiB)
17:38:06.126 INFO SparkContext - Created broadcast 264 from newAPIHadoopFile at PathSplitSource.java:96
17:38:06.144 INFO FileInputFormat - Total input files to process : 1
17:38:06.176 INFO SparkContext - Starting job: collect at ReadsSparkSinkUnitTest.java:182
17:38:06.177 INFO DAGScheduler - Got job 100 (collect at ReadsSparkSinkUnitTest.java:182) with 1 output partitions
17:38:06.177 INFO DAGScheduler - Final stage: ResultStage 141 (collect at ReadsSparkSinkUnitTest.java:182)
17:38:06.177 INFO DAGScheduler - Parents of final stage: List()
17:38:06.177 INFO DAGScheduler - Missing parents: List()
17:38:06.177 INFO DAGScheduler - Submitting ResultStage 141 (MapPartitionsRDD[656] at filter at ReadsSparkSource.java:96), which has no missing parents
17:38:06.192 INFO MemoryStore - Block broadcast_265 stored as values in memory (estimated size 426.2 KiB, free 1917.6 MiB)
17:38:06.193 INFO MemoryStore - Block broadcast_265_piece0 stored as bytes in memory (estimated size 153.6 KiB, free 1917.4 MiB)
17:38:06.194 INFO BlockManagerInfo - Added broadcast_265_piece0 in memory on localhost:37953 (size: 153.6 KiB, free: 1919.5 MiB)
17:38:06.194 INFO SparkContext - Created broadcast 265 from broadcast at DAGScheduler.scala:1580
17:38:06.194 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 141 (MapPartitionsRDD[656] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:38:06.194 INFO TaskSchedulerImpl - Adding task set 141.0 with 1 tasks resource profile 0
17:38:06.195 INFO TaskSetManager - Starting task 0.0 in stage 141.0 (TID 197) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7809 bytes)
17:38:06.195 INFO Executor - Running task 0.0 in stage 141.0 (TID 197)
17:38:06.226 INFO NewHadoopRDD - Input split: file:/tmp/ReadsSparkSinkUnitTest19999427906069391149.bam:0+238969
17:38:06.226 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:06.227 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:06.237 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:06.241 INFO Executor - Finished task 0.0 in stage 141.0 (TID 197). 651483 bytes result sent to driver
17:38:06.242 INFO TaskSetManager - Finished task 0.0 in stage 141.0 (TID 197) in 47 ms on localhost (executor driver) (1/1)
17:38:06.242 INFO TaskSchedulerImpl - Removed TaskSet 141.0, whose tasks have all completed, from pool
17:38:06.242 INFO DAGScheduler - ResultStage 141 (collect at ReadsSparkSinkUnitTest.java:182) finished in 0.065 s
17:38:06.243 INFO DAGScheduler - Job 100 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:06.243 INFO TaskSchedulerImpl - Killing all running tasks in stage 141: Stage finished
17:38:06.243 INFO DAGScheduler - Job 100 finished: collect at ReadsSparkSinkUnitTest.java:182, took 0.066211 s
17:38:06.252 INFO SparkContext - Starting job: count at ReadsSparkSinkUnitTest.java:185
17:38:06.253 INFO DAGScheduler - Got job 101 (count at ReadsSparkSinkUnitTest.java:185) with 1 output partitions
17:38:06.253 INFO DAGScheduler - Final stage: ResultStage 142 (count at ReadsSparkSinkUnitTest.java:185)
17:38:06.253 INFO DAGScheduler - Parents of final stage: List()
17:38:06.253 INFO DAGScheduler - Missing parents: List()
17:38:06.253 INFO DAGScheduler - Submitting ResultStage 142 (MapPartitionsRDD[637] at filter at ReadsSparkSource.java:96), which has no missing parents
17:38:06.268 INFO MemoryStore - Block broadcast_266 stored as values in memory (estimated size 426.1 KiB, free 1917.0 MiB)
17:38:06.269 INFO MemoryStore - Block broadcast_266_piece0 stored as bytes in memory (estimated size 153.6 KiB, free 1916.9 MiB)
17:38:06.269 INFO BlockManagerInfo - Added broadcast_266_piece0 in memory on localhost:37953 (size: 153.6 KiB, free: 1919.4 MiB)
17:38:06.269 INFO SparkContext - Created broadcast 266 from broadcast at DAGScheduler.scala:1580
17:38:06.270 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 142 (MapPartitionsRDD[637] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:38:06.270 INFO TaskSchedulerImpl - Adding task set 142.0 with 1 tasks resource profile 0
17:38:06.270 INFO TaskSetManager - Starting task 0.0 in stage 142.0 (TID 198) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7872 bytes)
17:38:06.270 INFO Executor - Running task 0.0 in stage 142.0 (TID 198)
17:38:06.300 INFO NewHadoopRDD - Input split: file:/home/runner/work/gatk/gatk/src/test/resources/org/broadinstitute/hellbender/tools/BQSR/HiSeq.1mb.1RG.2k_lines.bam:0+222075
17:38:06.301 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:06.302 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:06.309 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:06.313 INFO Executor - Finished task 0.0 in stage 142.0 (TID 198). 989 bytes result sent to driver
17:38:06.313 INFO TaskSetManager - Finished task 0.0 in stage 142.0 (TID 198) in 43 ms on localhost (executor driver) (1/1)
17:38:06.313 INFO TaskSchedulerImpl - Removed TaskSet 142.0, whose tasks have all completed, from pool
17:38:06.313 INFO DAGScheduler - ResultStage 142 (count at ReadsSparkSinkUnitTest.java:185) finished in 0.060 s
17:38:06.313 INFO DAGScheduler - Job 101 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:06.313 INFO TaskSchedulerImpl - Killing all running tasks in stage 142: Stage finished
17:38:06.313 INFO DAGScheduler - Job 101 finished: count at ReadsSparkSinkUnitTest.java:185, took 0.060762 s
17:38:06.316 INFO SparkContext - Starting job: count at ReadsSparkSinkUnitTest.java:185
17:38:06.316 INFO DAGScheduler - Got job 102 (count at ReadsSparkSinkUnitTest.java:185) with 1 output partitions
17:38:06.316 INFO DAGScheduler - Final stage: ResultStage 143 (count at ReadsSparkSinkUnitTest.java:185)
17:38:06.316 INFO DAGScheduler - Parents of final stage: List()
17:38:06.316 INFO DAGScheduler - Missing parents: List()
17:38:06.317 INFO DAGScheduler - Submitting ResultStage 143 (MapPartitionsRDD[656] at filter at ReadsSparkSource.java:96), which has no missing parents
17:38:06.332 INFO MemoryStore - Block broadcast_267 stored as values in memory (estimated size 426.1 KiB, free 1916.5 MiB)
17:38:06.336 INFO BlockManagerInfo - Removed broadcast_261_piece0 on localhost:37953 in memory (size: 9.6 KiB, free: 1919.4 MiB)
17:38:06.336 INFO MemoryStore - Block broadcast_267_piece0 stored as bytes in memory (estimated size 153.6 KiB, free 1916.5 MiB)
17:38:06.336 INFO BlockManagerInfo - Removed broadcast_266_piece0 on localhost:37953 in memory (size: 153.6 KiB, free: 1919.5 MiB)
17:38:06.336 INFO BlockManagerInfo - Added broadcast_267_piece0 in memory on localhost:37953 (size: 153.6 KiB, free: 1919.4 MiB)
17:38:06.336 INFO SparkContext - Created broadcast 267 from broadcast at DAGScheduler.scala:1580
17:38:06.337 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 143 (MapPartitionsRDD[656] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:38:06.337 INFO TaskSchedulerImpl - Adding task set 143.0 with 1 tasks resource profile 0
17:38:06.337 INFO BlockManagerInfo - Removed broadcast_262_piece0 on localhost:37953 in memory (size: 166.1 KiB, free: 1919.5 MiB)
17:38:06.337 INFO TaskSetManager - Starting task 0.0 in stage 143.0 (TID 199) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7809 bytes)
17:38:06.338 INFO BlockManagerInfo - Removed broadcast_265_piece0 on localhost:37953 in memory (size: 153.6 KiB, free: 1919.7 MiB)
17:38:06.338 INFO Executor - Running task 0.0 in stage 143.0 (TID 199)
17:38:06.339 INFO BlockManagerInfo - Removed broadcast_260_piece0 on localhost:37953 in memory (size: 9.6 KiB, free: 1919.7 MiB)
17:38:06.339 INFO BlockManagerInfo - Removed broadcast_263_piece0 on localhost:37953 in memory (size: 67.0 KiB, free: 1919.8 MiB)
17:38:06.371 INFO NewHadoopRDD - Input split: file:/tmp/ReadsSparkSinkUnitTest19999427906069391149.bam:0+238969
17:38:06.372 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:06.372 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:06.382 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:06.385 INFO Executor - Finished task 0.0 in stage 143.0 (TID 199). 989 bytes result sent to driver
17:38:06.386 INFO TaskSetManager - Finished task 0.0 in stage 143.0 (TID 199) in 49 ms on localhost (executor driver) (1/1)
17:38:06.386 INFO TaskSchedulerImpl - Removed TaskSet 143.0, whose tasks have all completed, from pool
17:38:06.386 INFO DAGScheduler - ResultStage 143 (count at ReadsSparkSinkUnitTest.java:185) finished in 0.069 s
17:38:06.386 INFO DAGScheduler - Job 102 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:06.386 INFO TaskSchedulerImpl - Killing all running tasks in stage 143: Stage finished
17:38:06.386 INFO DAGScheduler - Job 102 finished: count at ReadsSparkSinkUnitTest.java:185, took 0.070156 s
17:38:06.387 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:06.390 INFO MemoryStore - Block broadcast_268 stored as values in memory (estimated size 298.0 KiB, free 1918.5 MiB)
17:38:06.395 INFO MemoryStore - Block broadcast_268_piece0 stored as bytes in memory (estimated size 50.3 KiB, free 1918.4 MiB)
17:38:06.395 INFO BlockManagerInfo - Added broadcast_268_piece0 in memory on localhost:37953 (size: 50.3 KiB, free: 1919.7 MiB)
17:38:06.396 INFO SparkContext - Created broadcast 268 from newAPIHadoopFile at PathSplitSource.java:96
17:38:06.414 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:06.416 INFO MemoryStore - Block broadcast_269 stored as values in memory (estimated size 298.0 KiB, free 1918.1 MiB)
17:38:06.422 INFO MemoryStore - Block broadcast_269_piece0 stored as bytes in memory (estimated size 50.3 KiB, free 1918.1 MiB)
17:38:06.422 INFO BlockManagerInfo - Added broadcast_269_piece0 in memory on localhost:37953 (size: 50.3 KiB, free: 1919.7 MiB)
17:38:06.422 INFO SparkContext - Created broadcast 269 from newAPIHadoopFile at PathSplitSource.java:96
17:38:06.441 INFO FileInputFormat - Total input files to process : 1
17:38:06.443 INFO MemoryStore - Block broadcast_270 stored as values in memory (estimated size 160.7 KiB, free 1917.9 MiB)
17:38:06.443 INFO MemoryStore - Block broadcast_270_piece0 stored as bytes in memory (estimated size 9.6 KiB, free 1917.9 MiB)
17:38:06.444 INFO BlockManagerInfo - Added broadcast_270_piece0 in memory on localhost:37953 (size: 9.6 KiB, free: 1919.6 MiB)
17:38:06.444 INFO SparkContext - Created broadcast 270 from broadcast at ReadsSparkSink.java:133
17:38:06.445 INFO MemoryStore - Block broadcast_271 stored as values in memory (estimated size 163.2 KiB, free 1917.7 MiB)
17:38:06.446 INFO MemoryStore - Block broadcast_271_piece0 stored as bytes in memory (estimated size 9.6 KiB, free 1917.7 MiB)
17:38:06.446 INFO BlockManagerInfo - Added broadcast_271_piece0 in memory on localhost:37953 (size: 9.6 KiB, free: 1919.6 MiB)
17:38:06.446 INFO SparkContext - Created broadcast 271 from broadcast at BamSink.java:76
17:38:06.448 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:06.448 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:06.448 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:06.461 INFO SparkContext - Starting job: runJob at SparkHadoopWriter.scala:83
17:38:06.461 INFO DAGScheduler - Registering RDD 670 (mapToPair at SparkUtils.java:161) as input to shuffle 30
17:38:06.462 INFO DAGScheduler - Got job 103 (runJob at SparkHadoopWriter.scala:83) with 1 output partitions
17:38:06.462 INFO DAGScheduler - Final stage: ResultStage 145 (runJob at SparkHadoopWriter.scala:83)
17:38:06.462 INFO DAGScheduler - Parents of final stage: List(ShuffleMapStage 144)
17:38:06.462 INFO DAGScheduler - Missing parents: List(ShuffleMapStage 144)
17:38:06.462 INFO DAGScheduler - Submitting ShuffleMapStage 144 (MapPartitionsRDD[670] at mapToPair at SparkUtils.java:161), which has no missing parents
17:38:06.478 INFO MemoryStore - Block broadcast_272 stored as values in memory (estimated size 520.4 KiB, free 1917.2 MiB)
17:38:06.479 INFO MemoryStore - Block broadcast_272_piece0 stored as bytes in memory (estimated size 166.1 KiB, free 1917.1 MiB)
17:38:06.479 INFO BlockManagerInfo - Added broadcast_272_piece0 in memory on localhost:37953 (size: 166.1 KiB, free: 1919.5 MiB)
17:38:06.479 INFO SparkContext - Created broadcast 272 from broadcast at DAGScheduler.scala:1580
17:38:06.479 INFO DAGScheduler - Submitting 1 missing tasks from ShuffleMapStage 144 (MapPartitionsRDD[670] at mapToPair at SparkUtils.java:161) (first 15 tasks are for partitions Vector(0))
17:38:06.479 INFO TaskSchedulerImpl - Adding task set 144.0 with 1 tasks resource profile 0
17:38:06.480 INFO TaskSetManager - Starting task 0.0 in stage 144.0 (TID 200) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7901 bytes)
17:38:06.480 INFO Executor - Running task 0.0 in stage 144.0 (TID 200)
17:38:06.512 INFO NewHadoopRDD - Input split: file:/home/runner/work/gatk/gatk/src/test/resources/org/broadinstitute/hellbender/tools/BQSR/expected.HiSeq.1mb.1RG.2k_lines.alternate.recalibrated.DIQ.bam:0+216896
17:38:06.513 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:06.514 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:06.523 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:06.532 INFO Executor - Finished task 0.0 in stage 144.0 (TID 200). 1148 bytes result sent to driver
17:38:06.532 INFO TaskSetManager - Finished task 0.0 in stage 144.0 (TID 200) in 52 ms on localhost (executor driver) (1/1)
17:38:06.532 INFO TaskSchedulerImpl - Removed TaskSet 144.0, whose tasks have all completed, from pool
17:38:06.533 INFO DAGScheduler - ShuffleMapStage 144 (mapToPair at SparkUtils.java:161) finished in 0.071 s
17:38:06.533 INFO DAGScheduler - looking for newly runnable stages
17:38:06.533 INFO DAGScheduler - running: HashSet()
17:38:06.533 INFO DAGScheduler - waiting: HashSet(ResultStage 145)
17:38:06.533 INFO DAGScheduler - failed: HashSet()
17:38:06.533 INFO DAGScheduler - Submitting ResultStage 145 (MapPartitionsRDD[675] at mapToPair at BamSink.java:91), which has no missing parents
17:38:06.539 INFO MemoryStore - Block broadcast_273 stored as values in memory (estimated size 241.4 KiB, free 1916.8 MiB)
17:38:06.539 INFO MemoryStore - Block broadcast_273_piece0 stored as bytes in memory (estimated size 67.0 KiB, free 1916.8 MiB)
17:38:06.540 INFO BlockManagerInfo - Added broadcast_273_piece0 in memory on localhost:37953 (size: 67.0 KiB, free: 1919.4 MiB)
17:38:06.540 INFO SparkContext - Created broadcast 273 from broadcast at DAGScheduler.scala:1580
17:38:06.540 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 145 (MapPartitionsRDD[675] at mapToPair at BamSink.java:91) (first 15 tasks are for partitions Vector(0))
17:38:06.540 INFO TaskSchedulerImpl - Adding task set 145.0 with 1 tasks resource profile 0
17:38:06.541 INFO TaskSetManager - Starting task 0.0 in stage 145.0 (TID 201) (localhost, executor driver, partition 0, NODE_LOCAL, 7513 bytes)
17:38:06.541 INFO Executor - Running task 0.0 in stage 145.0 (TID 201)
17:38:06.546 INFO ShuffleBlockFetcherIterator - Getting 1 (343.8 KiB) non-empty blocks including 1 (343.8 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
17:38:06.546 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
17:38:06.557 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:06.557 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:06.557 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:06.557 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:06.557 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:06.557 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:06.565 WARN IntelDeflaterFactory - Intel Deflater not supported, using Java.util.zip.Deflater
17:38:06.580 INFO FileOutputCommitter - Saved output of task 'attempt_2025102717380613295550961020005_0675_r_000000_0' to file:/tmp/ReadsSparkSinkUnitTest23823218004399596999.bam.parts/_temporary/0/task_2025102717380613295550961020005_0675_r_000000
17:38:06.580 INFO SparkHadoopMapRedUtil - attempt_2025102717380613295550961020005_0675_r_000000_0: Committed. Elapsed time: 0 ms.
17:38:06.584 INFO Executor - Finished task 0.0 in stage 145.0 (TID 201). 1944 bytes result sent to driver
17:38:06.584 INFO BlockManagerInfo - Removed broadcast_267_piece0 on localhost:37953 in memory (size: 153.6 KiB, free: 1919.6 MiB)
17:38:06.584 INFO TaskSetManager - Finished task 0.0 in stage 145.0 (TID 201) in 43 ms on localhost (executor driver) (1/1)
17:38:06.585 INFO TaskSchedulerImpl - Removed TaskSet 145.0, whose tasks have all completed, from pool
17:38:06.585 INFO DAGScheduler - ResultStage 145 (runJob at SparkHadoopWriter.scala:83) finished in 0.052 s
17:38:06.585 INFO DAGScheduler - Job 103 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:06.585 INFO TaskSchedulerImpl - Killing all running tasks in stage 145: Stage finished
17:38:06.585 INFO DAGScheduler - Job 103 finished: runJob at SparkHadoopWriter.scala:83, took 0.123941 s
17:38:06.585 INFO BlockManagerInfo - Removed broadcast_272_piece0 on localhost:37953 in memory (size: 166.1 KiB, free: 1919.7 MiB)
17:38:06.585 INFO SparkHadoopWriter - Start to commit write Job job_2025102717380613295550961020005_0675.
17:38:06.586 INFO BlockManagerInfo - Removed broadcast_269_piece0 on localhost:37953 in memory (size: 50.3 KiB, free: 1919.8 MiB)
17:38:06.587 INFO BlockManagerInfo - Removed broadcast_264_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.8 MiB)
17:38:06.587 INFO BlockManagerInfo - Removed broadcast_258_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.9 MiB)
17:38:06.589 INFO SparkHadoopWriter - Write Job job_2025102717380613295550961020005_0675 committed. Elapsed time: 3 ms.
17:38:06.591 WARN IntelDeflaterFactory - Intel Deflater not supported, using Java.util.zip.Deflater
17:38:06.596 INFO HadoopFileSystemWrapper - Concatenating 3 parts to /tmp/ReadsSparkSinkUnitTest23823218004399596999.bam
17:38:06.600 INFO HadoopFileSystemWrapper - Concatenating to /tmp/ReadsSparkSinkUnitTest23823218004399596999.bam done
17:38:06.600 INFO IndexFileMerger - Merging .sbi files in temp directory /tmp/ReadsSparkSinkUnitTest23823218004399596999.bam.parts/ to /tmp/ReadsSparkSinkUnitTest23823218004399596999.bam.sbi
17:38:06.605 INFO IndexFileMerger - Done merging .sbi files
17:38:06.605 INFO IndexFileMerger - Merging .bai files in temp directory /tmp/ReadsSparkSinkUnitTest23823218004399596999.bam.parts/ to /tmp/ReadsSparkSinkUnitTest23823218004399596999.bam.bai
17:38:06.609 INFO IndexFileMerger - Done merging .bai files
17:38:06.610 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:06.611 INFO MemoryStore - Block broadcast_274 stored as values in memory (estimated size 320.0 B, free 1919.0 MiB)
17:38:06.611 INFO MemoryStore - Block broadcast_274_piece0 stored as bytes in memory (estimated size 233.0 B, free 1919.0 MiB)
17:38:06.611 INFO BlockManagerInfo - Added broadcast_274_piece0 in memory on localhost:37953 (size: 233.0 B, free: 1919.9 MiB)
17:38:06.612 INFO SparkContext - Created broadcast 274 from broadcast at BamSource.java:104
17:38:06.613 INFO MemoryStore - Block broadcast_275 stored as values in memory (estimated size 297.9 KiB, free 1918.7 MiB)
17:38:06.618 INFO MemoryStore - Block broadcast_275_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1918.7 MiB)
17:38:06.618 INFO BlockManagerInfo - Added broadcast_275_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.8 MiB)
17:38:06.618 INFO SparkContext - Created broadcast 275 from newAPIHadoopFile at PathSplitSource.java:96
17:38:06.627 INFO FileInputFormat - Total input files to process : 1
17:38:06.640 INFO SparkContext - Starting job: collect at ReadsSparkSinkUnitTest.java:182
17:38:06.640 INFO DAGScheduler - Got job 104 (collect at ReadsSparkSinkUnitTest.java:182) with 1 output partitions
17:38:06.640 INFO DAGScheduler - Final stage: ResultStage 146 (collect at ReadsSparkSinkUnitTest.java:182)
17:38:06.640 INFO DAGScheduler - Parents of final stage: List()
17:38:06.640 INFO DAGScheduler - Missing parents: List()
17:38:06.641 INFO DAGScheduler - Submitting ResultStage 146 (MapPartitionsRDD[681] at filter at ReadsSparkSource.java:96), which has no missing parents
17:38:06.646 INFO MemoryStore - Block broadcast_276 stored as values in memory (estimated size 148.2 KiB, free 1918.5 MiB)
17:38:06.647 INFO MemoryStore - Block broadcast_276_piece0 stored as bytes in memory (estimated size 54.6 KiB, free 1918.5 MiB)
17:38:06.647 INFO BlockManagerInfo - Added broadcast_276_piece0 in memory on localhost:37953 (size: 54.6 KiB, free: 1919.8 MiB)
17:38:06.647 INFO SparkContext - Created broadcast 276 from broadcast at DAGScheduler.scala:1580
17:38:06.647 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 146 (MapPartitionsRDD[681] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:38:06.647 INFO TaskSchedulerImpl - Adding task set 146.0 with 1 tasks resource profile 0
17:38:06.648 INFO TaskSetManager - Starting task 0.0 in stage 146.0 (TID 202) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7809 bytes)
17:38:06.648 INFO Executor - Running task 0.0 in stage 146.0 (TID 202)
17:38:06.660 INFO NewHadoopRDD - Input split: file:/tmp/ReadsSparkSinkUnitTest23823218004399596999.bam:0+236321
17:38:06.661 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:06.665 INFO Executor - Finished task 0.0 in stage 146.0 (TID 202). 650141 bytes result sent to driver
17:38:06.667 INFO TaskSetManager - Finished task 0.0 in stage 146.0 (TID 202) in 19 ms on localhost (executor driver) (1/1)
17:38:06.667 INFO TaskSchedulerImpl - Removed TaskSet 146.0, whose tasks have all completed, from pool
17:38:06.667 INFO DAGScheduler - ResultStage 146 (collect at ReadsSparkSinkUnitTest.java:182) finished in 0.026 s
17:38:06.667 INFO DAGScheduler - Job 104 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:06.667 INFO TaskSchedulerImpl - Killing all running tasks in stage 146: Stage finished
17:38:06.667 INFO DAGScheduler - Job 104 finished: collect at ReadsSparkSinkUnitTest.java:182, took 0.026948 s
17:38:06.677 INFO SparkContext - Starting job: count at ReadsSparkSinkUnitTest.java:185
17:38:06.677 INFO DAGScheduler - Got job 105 (count at ReadsSparkSinkUnitTest.java:185) with 1 output partitions
17:38:06.677 INFO DAGScheduler - Final stage: ResultStage 147 (count at ReadsSparkSinkUnitTest.java:185)
17:38:06.677 INFO DAGScheduler - Parents of final stage: List()
17:38:06.677 INFO DAGScheduler - Missing parents: List()
17:38:06.677 INFO DAGScheduler - Submitting ResultStage 147 (MapPartitionsRDD[663] at filter at ReadsSparkSource.java:96), which has no missing parents
17:38:06.692 INFO MemoryStore - Block broadcast_277 stored as values in memory (estimated size 426.1 KiB, free 1918.1 MiB)
17:38:06.694 INFO MemoryStore - Block broadcast_277_piece0 stored as bytes in memory (estimated size 153.6 KiB, free 1917.9 MiB)
17:38:06.694 INFO BlockManagerInfo - Added broadcast_277_piece0 in memory on localhost:37953 (size: 153.6 KiB, free: 1919.6 MiB)
17:38:06.694 INFO SparkContext - Created broadcast 277 from broadcast at DAGScheduler.scala:1580
17:38:06.694 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 147 (MapPartitionsRDD[663] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:38:06.694 INFO TaskSchedulerImpl - Adding task set 147.0 with 1 tasks resource profile 0
17:38:06.695 INFO TaskSetManager - Starting task 0.0 in stage 147.0 (TID 203) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7912 bytes)
17:38:06.695 INFO Executor - Running task 0.0 in stage 147.0 (TID 203)
17:38:06.725 INFO NewHadoopRDD - Input split: file:/home/runner/work/gatk/gatk/src/test/resources/org/broadinstitute/hellbender/tools/BQSR/expected.HiSeq.1mb.1RG.2k_lines.alternate.recalibrated.DIQ.bam:0+216896
17:38:06.726 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:06.727 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:06.736 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:06.740 INFO Executor - Finished task 0.0 in stage 147.0 (TID 203). 989 bytes result sent to driver
17:38:06.740 INFO TaskSetManager - Finished task 0.0 in stage 147.0 (TID 203) in 46 ms on localhost (executor driver) (1/1)
17:38:06.740 INFO TaskSchedulerImpl - Removed TaskSet 147.0, whose tasks have all completed, from pool
17:38:06.741 INFO DAGScheduler - ResultStage 147 (count at ReadsSparkSinkUnitTest.java:185) finished in 0.064 s
17:38:06.741 INFO DAGScheduler - Job 105 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:06.741 INFO TaskSchedulerImpl - Killing all running tasks in stage 147: Stage finished
17:38:06.741 INFO DAGScheduler - Job 105 finished: count at ReadsSparkSinkUnitTest.java:185, took 0.064010 s
17:38:06.744 INFO SparkContext - Starting job: count at ReadsSparkSinkUnitTest.java:185
17:38:06.744 INFO DAGScheduler - Got job 106 (count at ReadsSparkSinkUnitTest.java:185) with 1 output partitions
17:38:06.744 INFO DAGScheduler - Final stage: ResultStage 148 (count at ReadsSparkSinkUnitTest.java:185)
17:38:06.744 INFO DAGScheduler - Parents of final stage: List()
17:38:06.744 INFO DAGScheduler - Missing parents: List()
17:38:06.744 INFO DAGScheduler - Submitting ResultStage 148 (MapPartitionsRDD[681] at filter at ReadsSparkSource.java:96), which has no missing parents
17:38:06.750 INFO MemoryStore - Block broadcast_278 stored as values in memory (estimated size 148.1 KiB, free 1917.8 MiB)
17:38:06.751 INFO MemoryStore - Block broadcast_278_piece0 stored as bytes in memory (estimated size 54.5 KiB, free 1917.7 MiB)
17:38:06.751 INFO BlockManagerInfo - Added broadcast_278_piece0 in memory on localhost:37953 (size: 54.5 KiB, free: 1919.6 MiB)
17:38:06.751 INFO SparkContext - Created broadcast 278 from broadcast at DAGScheduler.scala:1580
17:38:06.751 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 148 (MapPartitionsRDD[681] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:38:06.751 INFO TaskSchedulerImpl - Adding task set 148.0 with 1 tasks resource profile 0
17:38:06.752 INFO TaskSetManager - Starting task 0.0 in stage 148.0 (TID 204) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7809 bytes)
17:38:06.752 INFO Executor - Running task 0.0 in stage 148.0 (TID 204)
17:38:06.763 INFO NewHadoopRDD - Input split: file:/tmp/ReadsSparkSinkUnitTest23823218004399596999.bam:0+236321
17:38:06.764 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:06.768 INFO Executor - Finished task 0.0 in stage 148.0 (TID 204). 989 bytes result sent to driver
17:38:06.768 INFO TaskSetManager - Finished task 0.0 in stage 148.0 (TID 204) in 17 ms on localhost (executor driver) (1/1)
17:38:06.768 INFO TaskSchedulerImpl - Removed TaskSet 148.0, whose tasks have all completed, from pool
17:38:06.768 INFO DAGScheduler - ResultStage 148 (count at ReadsSparkSinkUnitTest.java:185) finished in 0.023 s
17:38:06.768 INFO DAGScheduler - Job 106 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:06.768 INFO TaskSchedulerImpl - Killing all running tasks in stage 148: Stage finished
17:38:06.768 INFO DAGScheduler - Job 106 finished: count at ReadsSparkSinkUnitTest.java:185, took 0.024267 s
17:38:06.769 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:06.771 INFO MemoryStore - Block broadcast_279 stored as values in memory (estimated size 298.0 KiB, free 1917.4 MiB)
17:38:06.776 INFO MemoryStore - Block broadcast_279_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1917.4 MiB)
17:38:06.776 INFO BlockManagerInfo - Added broadcast_279_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.5 MiB)
17:38:06.777 INFO SparkContext - Created broadcast 279 from newAPIHadoopFile at PathSplitSource.java:96
17:38:06.796 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:06.798 INFO MemoryStore - Block broadcast_280 stored as values in memory (estimated size 298.0 KiB, free 1917.1 MiB)
17:38:06.803 INFO BlockManagerInfo - Removed broadcast_275_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.6 MiB)
17:38:06.803 INFO BlockManagerInfo - Removed broadcast_278_piece0 on localhost:37953 in memory (size: 54.5 KiB, free: 1919.6 MiB)
17:38:06.804 INFO BlockManagerInfo - Removed broadcast_273_piece0 on localhost:37953 in memory (size: 67.0 KiB, free: 1919.7 MiB)
17:38:06.805 INFO BlockManagerInfo - Removed broadcast_277_piece0 on localhost:37953 in memory (size: 153.6 KiB, free: 1919.8 MiB)
17:38:06.805 INFO BlockManagerInfo - Removed broadcast_268_piece0 on localhost:37953 in memory (size: 50.3 KiB, free: 1919.9 MiB)
17:38:06.805 INFO BlockManagerInfo - Removed broadcast_271_piece0 on localhost:37953 in memory (size: 9.6 KiB, free: 1919.9 MiB)
17:38:06.806 INFO BlockManagerInfo - Removed broadcast_276_piece0 on localhost:37953 in memory (size: 54.6 KiB, free: 1919.9 MiB)
17:38:06.806 INFO MemoryStore - Block broadcast_280_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1919.2 MiB)
17:38:06.806 INFO BlockManagerInfo - Removed broadcast_270_piece0 on localhost:37953 in memory (size: 9.6 KiB, free: 1920.0 MiB)
17:38:06.806 INFO BlockManagerInfo - Added broadcast_280_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.9 MiB)
17:38:06.807 INFO SparkContext - Created broadcast 280 from newAPIHadoopFile at PathSplitSource.java:96
17:38:06.807 INFO BlockManagerInfo - Removed broadcast_274_piece0 on localhost:37953 in memory (size: 233.0 B, free: 1919.9 MiB)
17:38:06.825 INFO FileInputFormat - Total input files to process : 1
17:38:06.827 INFO MemoryStore - Block broadcast_281 stored as values in memory (estimated size 19.6 KiB, free 1919.3 MiB)
17:38:06.827 INFO MemoryStore - Block broadcast_281_piece0 stored as bytes in memory (estimated size 1890.0 B, free 1919.3 MiB)
17:38:06.827 INFO BlockManagerInfo - Added broadcast_281_piece0 in memory on localhost:37953 (size: 1890.0 B, free: 1919.9 MiB)
17:38:06.828 INFO SparkContext - Created broadcast 281 from broadcast at ReadsSparkSink.java:133
17:38:06.828 INFO MemoryStore - Block broadcast_282 stored as values in memory (estimated size 20.0 KiB, free 1919.3 MiB)
17:38:06.829 INFO MemoryStore - Block broadcast_282_piece0 stored as bytes in memory (estimated size 1890.0 B, free 1919.3 MiB)
17:38:06.829 INFO BlockManagerInfo - Added broadcast_282_piece0 in memory on localhost:37953 (size: 1890.0 B, free: 1919.9 MiB)
17:38:06.829 INFO SparkContext - Created broadcast 282 from broadcast at BamSink.java:76
17:38:06.831 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:06.831 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:06.831 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:06.844 INFO SparkContext - Starting job: runJob at SparkHadoopWriter.scala:83
17:38:06.845 INFO DAGScheduler - Registering RDD 695 (mapToPair at SparkUtils.java:161) as input to shuffle 31
17:38:06.845 INFO DAGScheduler - Got job 107 (runJob at SparkHadoopWriter.scala:83) with 1 output partitions
17:38:06.845 INFO DAGScheduler - Final stage: ResultStage 150 (runJob at SparkHadoopWriter.scala:83)
17:38:06.845 INFO DAGScheduler - Parents of final stage: List(ShuffleMapStage 149)
17:38:06.845 INFO DAGScheduler - Missing parents: List(ShuffleMapStage 149)
17:38:06.845 INFO DAGScheduler - Submitting ShuffleMapStage 149 (MapPartitionsRDD[695] at mapToPair at SparkUtils.java:161), which has no missing parents
17:38:06.861 INFO MemoryStore - Block broadcast_283 stored as values in memory (estimated size 434.3 KiB, free 1918.9 MiB)
17:38:06.862 INFO MemoryStore - Block broadcast_283_piece0 stored as bytes in memory (estimated size 157.6 KiB, free 1918.7 MiB)
17:38:06.862 INFO BlockManagerInfo - Added broadcast_283_piece0 in memory on localhost:37953 (size: 157.6 KiB, free: 1919.7 MiB)
17:38:06.862 INFO SparkContext - Created broadcast 283 from broadcast at DAGScheduler.scala:1580
17:38:06.862 INFO DAGScheduler - Submitting 1 missing tasks from ShuffleMapStage 149 (MapPartitionsRDD[695] at mapToPair at SparkUtils.java:161) (first 15 tasks are for partitions Vector(0))
17:38:06.862 INFO TaskSchedulerImpl - Adding task set 149.0 with 1 tasks resource profile 0
17:38:06.863 INFO TaskSetManager - Starting task 0.0 in stage 149.0 (TID 205) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7882 bytes)
17:38:06.863 INFO Executor - Running task 0.0 in stage 149.0 (TID 205)
17:38:06.895 INFO NewHadoopRDD - Input split: file:/home/runner/work/gatk/gatk/src/test/resources/org/broadinstitute/hellbender/tools/BQSR/CEUTrio.HiSeq.WGS.b37.ch20.1m-1m1k.NA12878.bam:0+211123
17:38:06.896 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:06.897 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:06.902 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:06.910 INFO Executor - Finished task 0.0 in stage 149.0 (TID 205). 1148 bytes result sent to driver
17:38:06.911 INFO TaskSetManager - Finished task 0.0 in stage 149.0 (TID 205) in 48 ms on localhost (executor driver) (1/1)
17:38:06.911 INFO DAGScheduler - ShuffleMapStage 149 (mapToPair at SparkUtils.java:161) finished in 0.066 s
17:38:06.911 INFO DAGScheduler - looking for newly runnable stages
17:38:06.911 INFO DAGScheduler - running: HashSet()
17:38:06.911 INFO DAGScheduler - waiting: HashSet(ResultStage 150)
17:38:06.911 INFO DAGScheduler - failed: HashSet()
17:38:06.911 INFO DAGScheduler - Submitting ResultStage 150 (MapPartitionsRDD[700] at mapToPair at BamSink.java:91), which has no missing parents
17:38:06.912 INFO TaskSchedulerImpl - Removed TaskSet 149.0, whose tasks have all completed, from pool
17:38:06.917 INFO MemoryStore - Block broadcast_284 stored as values in memory (estimated size 155.3 KiB, free 1918.5 MiB)
17:38:06.918 INFO MemoryStore - Block broadcast_284_piece0 stored as bytes in memory (estimated size 58.4 KiB, free 1918.5 MiB)
17:38:06.918 INFO BlockManagerInfo - Added broadcast_284_piece0 in memory on localhost:37953 (size: 58.4 KiB, free: 1919.7 MiB)
17:38:06.918 INFO SparkContext - Created broadcast 284 from broadcast at DAGScheduler.scala:1580
17:38:06.918 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 150 (MapPartitionsRDD[700] at mapToPair at BamSink.java:91) (first 15 tasks are for partitions Vector(0))
17:38:06.918 INFO TaskSchedulerImpl - Adding task set 150.0 with 1 tasks resource profile 0
17:38:06.919 INFO TaskSetManager - Starting task 0.0 in stage 150.0 (TID 206) (localhost, executor driver, partition 0, NODE_LOCAL, 7513 bytes)
17:38:06.919 INFO Executor - Running task 0.0 in stage 150.0 (TID 206)
17:38:06.924 INFO ShuffleBlockFetcherIterator - Getting 1 (312.6 KiB) non-empty blocks including 1 (312.6 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
17:38:06.924 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
17:38:06.935 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:06.936 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:06.936 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:06.936 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:06.936 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:06.936 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:06.943 WARN IntelDeflaterFactory - Intel Deflater not supported, using Java.util.zip.Deflater
17:38:06.959 INFO FileOutputCommitter - Saved output of task 'attempt_202510271738062774354933822357681_0700_r_000000_0' to file:/tmp/ReadsSparkSinkUnitTest32990010356387330928.bam.parts/_temporary/0/task_202510271738062774354933822357681_0700_r_000000
17:38:06.959 INFO SparkHadoopMapRedUtil - attempt_202510271738062774354933822357681_0700_r_000000_0: Committed. Elapsed time: 0 ms.
17:38:06.959 INFO Executor - Finished task 0.0 in stage 150.0 (TID 206). 1858 bytes result sent to driver
17:38:06.960 INFO TaskSetManager - Finished task 0.0 in stage 150.0 (TID 206) in 41 ms on localhost (executor driver) (1/1)
17:38:06.960 INFO TaskSchedulerImpl - Removed TaskSet 150.0, whose tasks have all completed, from pool
17:38:06.960 INFO DAGScheduler - ResultStage 150 (runJob at SparkHadoopWriter.scala:83) finished in 0.048 s
17:38:06.960 INFO DAGScheduler - Job 107 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:06.960 INFO TaskSchedulerImpl - Killing all running tasks in stage 150: Stage finished
17:38:06.960 INFO DAGScheduler - Job 107 finished: runJob at SparkHadoopWriter.scala:83, took 0.115889 s
17:38:06.960 INFO SparkHadoopWriter - Start to commit write Job job_202510271738062774354933822357681_0700.
17:38:06.964 INFO SparkHadoopWriter - Write Job job_202510271738062774354933822357681_0700 committed. Elapsed time: 3 ms.
17:38:06.966 WARN IntelDeflaterFactory - Intel Deflater not supported, using Java.util.zip.Deflater
17:38:06.971 INFO HadoopFileSystemWrapper - Concatenating 3 parts to /tmp/ReadsSparkSinkUnitTest32990010356387330928.bam
17:38:06.975 INFO HadoopFileSystemWrapper - Concatenating to /tmp/ReadsSparkSinkUnitTest32990010356387330928.bam done
17:38:06.975 INFO IndexFileMerger - Merging .sbi files in temp directory /tmp/ReadsSparkSinkUnitTest32990010356387330928.bam.parts/ to /tmp/ReadsSparkSinkUnitTest32990010356387330928.bam.sbi
17:38:06.979 INFO IndexFileMerger - Done merging .sbi files
17:38:06.979 INFO IndexFileMerger - Merging .bai files in temp directory /tmp/ReadsSparkSinkUnitTest32990010356387330928.bam.parts/ to /tmp/ReadsSparkSinkUnitTest32990010356387330928.bam.bai
17:38:06.983 INFO IndexFileMerger - Done merging .bai files
17:38:06.983 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:06.984 INFO MemoryStore - Block broadcast_285 stored as values in memory (estimated size 312.0 B, free 1918.5 MiB)
17:38:06.985 INFO MemoryStore - Block broadcast_285_piece0 stored as bytes in memory (estimated size 231.0 B, free 1918.5 MiB)
17:38:06.985 INFO BlockManagerInfo - Added broadcast_285_piece0 in memory on localhost:37953 (size: 231.0 B, free: 1919.7 MiB)
17:38:06.985 INFO SparkContext - Created broadcast 285 from broadcast at BamSource.java:104
17:38:06.986 INFO MemoryStore - Block broadcast_286 stored as values in memory (estimated size 297.9 KiB, free 1918.2 MiB)
17:38:06.992 INFO MemoryStore - Block broadcast_286_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1918.2 MiB)
17:38:06.992 INFO BlockManagerInfo - Added broadcast_286_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.6 MiB)
17:38:06.992 INFO SparkContext - Created broadcast 286 from newAPIHadoopFile at PathSplitSource.java:96
17:38:07.000 INFO FileInputFormat - Total input files to process : 1
17:38:07.013 INFO SparkContext - Starting job: collect at ReadsSparkSinkUnitTest.java:182
17:38:07.014 INFO DAGScheduler - Got job 108 (collect at ReadsSparkSinkUnitTest.java:182) with 1 output partitions
17:38:07.014 INFO DAGScheduler - Final stage: ResultStage 151 (collect at ReadsSparkSinkUnitTest.java:182)
17:38:07.014 INFO DAGScheduler - Parents of final stage: List()
17:38:07.014 INFO DAGScheduler - Missing parents: List()
17:38:07.014 INFO DAGScheduler - Submitting ResultStage 151 (MapPartitionsRDD[706] at filter at ReadsSparkSource.java:96), which has no missing parents
17:38:07.019 INFO MemoryStore - Block broadcast_287 stored as values in memory (estimated size 148.2 KiB, free 1918.0 MiB)
17:38:07.023 INFO MemoryStore - Block broadcast_287_piece0 stored as bytes in memory (estimated size 54.6 KiB, free 1918.1 MiB)
17:38:07.023 INFO BlockManagerInfo - Removed broadcast_283_piece0 on localhost:37953 in memory (size: 157.6 KiB, free: 1919.8 MiB)
17:38:07.023 INFO BlockManagerInfo - Added broadcast_287_piece0 in memory on localhost:37953 (size: 54.6 KiB, free: 1919.7 MiB)
17:38:07.023 INFO SparkContext - Created broadcast 287 from broadcast at DAGScheduler.scala:1580
17:38:07.023 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 151 (MapPartitionsRDD[706] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:38:07.023 INFO TaskSchedulerImpl - Adding task set 151.0 with 1 tasks resource profile 0
17:38:07.023 INFO BlockManagerInfo - Removed broadcast_280_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.8 MiB)
17:38:07.024 INFO TaskSetManager - Starting task 0.0 in stage 151.0 (TID 207) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7809 bytes)
17:38:07.024 INFO Executor - Running task 0.0 in stage 151.0 (TID 207)
17:38:07.025 INFO BlockManagerInfo - Removed broadcast_284_piece0 on localhost:37953 in memory (size: 58.4 KiB, free: 1919.8 MiB)
17:38:07.025 INFO BlockManagerInfo - Removed broadcast_281_piece0 on localhost:37953 in memory (size: 1890.0 B, free: 1919.8 MiB)
17:38:07.026 INFO BlockManagerInfo - Removed broadcast_282_piece0 on localhost:37953 in memory (size: 1890.0 B, free: 1919.8 MiB)
17:38:07.036 INFO NewHadoopRDD - Input split: file:/tmp/ReadsSparkSinkUnitTest32990010356387330928.bam:0+237078
17:38:07.037 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:07.041 INFO Executor - Finished task 0.0 in stage 151.0 (TID 207). 749470 bytes result sent to driver
17:38:07.042 INFO TaskSetManager - Finished task 0.0 in stage 151.0 (TID 207) in 18 ms on localhost (executor driver) (1/1)
17:38:07.042 INFO TaskSchedulerImpl - Removed TaskSet 151.0, whose tasks have all completed, from pool
17:38:07.042 INFO DAGScheduler - ResultStage 151 (collect at ReadsSparkSinkUnitTest.java:182) finished in 0.028 s
17:38:07.042 INFO DAGScheduler - Job 108 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:07.042 INFO TaskSchedulerImpl - Killing all running tasks in stage 151: Stage finished
17:38:07.042 INFO DAGScheduler - Job 108 finished: collect at ReadsSparkSinkUnitTest.java:182, took 0.029122 s
17:38:07.053 INFO SparkContext - Starting job: count at ReadsSparkSinkUnitTest.java:185
17:38:07.054 INFO DAGScheduler - Got job 109 (count at ReadsSparkSinkUnitTest.java:185) with 1 output partitions
17:38:07.054 INFO DAGScheduler - Final stage: ResultStage 152 (count at ReadsSparkSinkUnitTest.java:185)
17:38:07.054 INFO DAGScheduler - Parents of final stage: List()
17:38:07.054 INFO DAGScheduler - Missing parents: List()
17:38:07.054 INFO DAGScheduler - Submitting ResultStage 152 (MapPartitionsRDD[688] at filter at ReadsSparkSource.java:96), which has no missing parents
17:38:07.069 INFO MemoryStore - Block broadcast_288 stored as values in memory (estimated size 426.1 KiB, free 1918.7 MiB)
17:38:07.070 INFO MemoryStore - Block broadcast_288_piece0 stored as bytes in memory (estimated size 153.6 KiB, free 1918.6 MiB)
17:38:07.070 INFO BlockManagerInfo - Added broadcast_288_piece0 in memory on localhost:37953 (size: 153.6 KiB, free: 1919.7 MiB)
17:38:07.071 INFO SparkContext - Created broadcast 288 from broadcast at DAGScheduler.scala:1580
17:38:07.071 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 152 (MapPartitionsRDD[688] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:38:07.071 INFO TaskSchedulerImpl - Adding task set 152.0 with 1 tasks resource profile 0
17:38:07.071 INFO TaskSetManager - Starting task 0.0 in stage 152.0 (TID 208) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7893 bytes)
17:38:07.072 INFO Executor - Running task 0.0 in stage 152.0 (TID 208)
17:38:07.102 INFO NewHadoopRDD - Input split: file:/home/runner/work/gatk/gatk/src/test/resources/org/broadinstitute/hellbender/tools/BQSR/CEUTrio.HiSeq.WGS.b37.ch20.1m-1m1k.NA12878.bam:0+211123
17:38:07.103 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:07.104 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:07.109 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:07.112 INFO Executor - Finished task 0.0 in stage 152.0 (TID 208). 989 bytes result sent to driver
17:38:07.113 INFO TaskSetManager - Finished task 0.0 in stage 152.0 (TID 208) in 42 ms on localhost (executor driver) (1/1)
17:38:07.113 INFO TaskSchedulerImpl - Removed TaskSet 152.0, whose tasks have all completed, from pool
17:38:07.113 INFO DAGScheduler - ResultStage 152 (count at ReadsSparkSinkUnitTest.java:185) finished in 0.059 s
17:38:07.113 INFO DAGScheduler - Job 109 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:07.113 INFO TaskSchedulerImpl - Killing all running tasks in stage 152: Stage finished
17:38:07.113 INFO DAGScheduler - Job 109 finished: count at ReadsSparkSinkUnitTest.java:185, took 0.059622 s
17:38:07.116 INFO SparkContext - Starting job: count at ReadsSparkSinkUnitTest.java:185
17:38:07.116 INFO DAGScheduler - Got job 110 (count at ReadsSparkSinkUnitTest.java:185) with 1 output partitions
17:38:07.116 INFO DAGScheduler - Final stage: ResultStage 153 (count at ReadsSparkSinkUnitTest.java:185)
17:38:07.116 INFO DAGScheduler - Parents of final stage: List()
17:38:07.117 INFO DAGScheduler - Missing parents: List()
17:38:07.117 INFO DAGScheduler - Submitting ResultStage 153 (MapPartitionsRDD[706] at filter at ReadsSparkSource.java:96), which has no missing parents
17:38:07.122 INFO MemoryStore - Block broadcast_289 stored as values in memory (estimated size 148.1 KiB, free 1918.4 MiB)
17:38:07.123 INFO MemoryStore - Block broadcast_289_piece0 stored as bytes in memory (estimated size 54.5 KiB, free 1918.4 MiB)
17:38:07.123 INFO BlockManagerInfo - Added broadcast_289_piece0 in memory on localhost:37953 (size: 54.5 KiB, free: 1919.6 MiB)
17:38:07.123 INFO SparkContext - Created broadcast 289 from broadcast at DAGScheduler.scala:1580
17:38:07.123 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 153 (MapPartitionsRDD[706] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:38:07.123 INFO TaskSchedulerImpl - Adding task set 153.0 with 1 tasks resource profile 0
17:38:07.124 INFO TaskSetManager - Starting task 0.0 in stage 153.0 (TID 209) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7809 bytes)
17:38:07.124 INFO Executor - Running task 0.0 in stage 153.0 (TID 209)
17:38:07.135 INFO NewHadoopRDD - Input split: file:/tmp/ReadsSparkSinkUnitTest32990010356387330928.bam:0+237078
17:38:07.135 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:07.138 INFO Executor - Finished task 0.0 in stage 153.0 (TID 209). 989 bytes result sent to driver
17:38:07.139 INFO TaskSetManager - Finished task 0.0 in stage 153.0 (TID 209) in 15 ms on localhost (executor driver) (1/1)
17:38:07.139 INFO TaskSchedulerImpl - Removed TaskSet 153.0, whose tasks have all completed, from pool
17:38:07.139 INFO DAGScheduler - ResultStage 153 (count at ReadsSparkSinkUnitTest.java:185) finished in 0.022 s
17:38:07.139 INFO DAGScheduler - Job 110 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:07.139 INFO TaskSchedulerImpl - Killing all running tasks in stage 153: Stage finished
17:38:07.139 INFO DAGScheduler - Job 110 finished: count at ReadsSparkSinkUnitTest.java:185, took 0.022661 s
17:38:07.141 INFO MemoryStore - Block broadcast_290 stored as values in memory (estimated size 576.0 B, free 1918.4 MiB)
17:38:07.142 INFO MemoryStore - Block broadcast_290_piece0 stored as bytes in memory (estimated size 228.0 B, free 1918.4 MiB)
17:38:07.142 INFO BlockManagerInfo - Added broadcast_290_piece0 in memory on localhost:37953 (size: 228.0 B, free: 1919.6 MiB)
17:38:07.142 INFO SparkContext - Created broadcast 290 from broadcast at CramSource.java:114
17:38:07.143 INFO MemoryStore - Block broadcast_291 stored as values in memory (estimated size 297.9 KiB, free 1918.1 MiB)
17:38:07.148 INFO MemoryStore - Block broadcast_291_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1918.0 MiB)
17:38:07.148 INFO BlockManagerInfo - Added broadcast_291_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.6 MiB)
17:38:07.149 INFO SparkContext - Created broadcast 291 from newAPIHadoopFile at PathSplitSource.java:96
17:38:07.163 INFO MemoryStore - Block broadcast_292 stored as values in memory (estimated size 576.0 B, free 1918.0 MiB)
17:38:07.164 INFO MemoryStore - Block broadcast_292_piece0 stored as bytes in memory (estimated size 228.0 B, free 1918.0 MiB)
17:38:07.164 INFO BlockManagerInfo - Added broadcast_292_piece0 in memory on localhost:37953 (size: 228.0 B, free: 1919.6 MiB)
17:38:07.164 INFO SparkContext - Created broadcast 292 from broadcast at CramSource.java:114
17:38:07.165 INFO MemoryStore - Block broadcast_293 stored as values in memory (estimated size 297.9 KiB, free 1917.7 MiB)
17:38:07.170 INFO MemoryStore - Block broadcast_293_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1917.7 MiB)
17:38:07.170 INFO BlockManagerInfo - Added broadcast_293_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.5 MiB)
17:38:07.171 INFO SparkContext - Created broadcast 293 from newAPIHadoopFile at PathSplitSource.java:96
17:38:07.184 INFO FileInputFormat - Total input files to process : 1
17:38:07.185 INFO MemoryStore - Block broadcast_294 stored as values in memory (estimated size 6.0 KiB, free 1917.7 MiB)
17:38:07.188 INFO MemoryStore - Block broadcast_294_piece0 stored as bytes in memory (estimated size 1473.0 B, free 1917.7 MiB)
17:38:07.188 INFO BlockManagerInfo - Added broadcast_294_piece0 in memory on localhost:37953 (size: 1473.0 B, free: 1919.5 MiB)
17:38:07.188 INFO BlockManagerInfo - Removed broadcast_293_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.6 MiB)
17:38:07.188 INFO SparkContext - Created broadcast 294 from broadcast at ReadsSparkSink.java:133
17:38:07.189 INFO BlockManagerInfo - Removed broadcast_289_piece0 on localhost:37953 in memory (size: 54.5 KiB, free: 1919.6 MiB)
17:38:07.189 INFO MemoryStore - Block broadcast_295 stored as values in memory (estimated size 6.2 KiB, free 1918.2 MiB)
17:38:07.189 INFO MemoryStore - Block broadcast_295_piece0 stored as bytes in memory (estimated size 1473.0 B, free 1918.3 MiB)
17:38:07.189 INFO BlockManagerInfo - Removed broadcast_287_piece0 on localhost:37953 in memory (size: 54.6 KiB, free: 1919.7 MiB)
17:38:07.189 INFO BlockManagerInfo - Added broadcast_295_piece0 in memory on localhost:37953 (size: 1473.0 B, free: 1919.7 MiB)
17:38:07.190 INFO SparkContext - Created broadcast 295 from broadcast at CramSink.java:76
17:38:07.190 INFO BlockManagerInfo - Removed broadcast_286_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.7 MiB)
17:38:07.190 INFO BlockManagerInfo - Removed broadcast_292_piece0 on localhost:37953 in memory (size: 228.0 B, free: 1919.7 MiB)
17:38:07.191 INFO BlockManagerInfo - Removed broadcast_285_piece0 on localhost:37953 in memory (size: 231.0 B, free: 1919.7 MiB)
17:38:07.191 INFO BlockManagerInfo - Removed broadcast_288_piece0 on localhost:37953 in memory (size: 153.6 KiB, free: 1919.9 MiB)
17:38:07.191 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:07.192 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:07.192 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:07.192 INFO BlockManagerInfo - Removed broadcast_279_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.9 MiB)
17:38:07.204 INFO SparkContext - Starting job: runJob at SparkHadoopWriter.scala:83
17:38:07.205 INFO DAGScheduler - Registering RDD 718 (mapToPair at SparkUtils.java:161) as input to shuffle 32
17:38:07.205 INFO DAGScheduler - Got job 111 (runJob at SparkHadoopWriter.scala:83) with 1 output partitions
17:38:07.205 INFO DAGScheduler - Final stage: ResultStage 155 (runJob at SparkHadoopWriter.scala:83)
17:38:07.205 INFO DAGScheduler - Parents of final stage: List(ShuffleMapStage 154)
17:38:07.205 INFO DAGScheduler - Missing parents: List(ShuffleMapStage 154)
17:38:07.205 INFO DAGScheduler - Submitting ShuffleMapStage 154 (MapPartitionsRDD[718] at mapToPair at SparkUtils.java:161), which has no missing parents
17:38:07.215 INFO MemoryStore - Block broadcast_296 stored as values in memory (estimated size 292.8 KiB, free 1919.4 MiB)
17:38:07.216 INFO MemoryStore - Block broadcast_296_piece0 stored as bytes in memory (estimated size 107.3 KiB, free 1919.3 MiB)
17:38:07.216 INFO BlockManagerInfo - Added broadcast_296_piece0 in memory on localhost:37953 (size: 107.3 KiB, free: 1919.8 MiB)
17:38:07.217 INFO SparkContext - Created broadcast 296 from broadcast at DAGScheduler.scala:1580
17:38:07.217 INFO DAGScheduler - Submitting 1 missing tasks from ShuffleMapStage 154 (MapPartitionsRDD[718] at mapToPair at SparkUtils.java:161) (first 15 tasks are for partitions Vector(0))
17:38:07.217 INFO TaskSchedulerImpl - Adding task set 154.0 with 1 tasks resource profile 0
17:38:07.217 INFO TaskSetManager - Starting task 0.0 in stage 154.0 (TID 210) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7869 bytes)
17:38:07.218 INFO Executor - Running task 0.0 in stage 154.0 (TID 210)
17:38:07.239 INFO NewHadoopRDD - Input split: file:/home/runner/work/gatk/gatk/src/test/resources/org/broadinstitute/hellbender/tools/BQSR/NA12878.chr17_69k_70k.dictFix.cram:0+50619
17:38:07.249 INFO Executor - Finished task 0.0 in stage 154.0 (TID 210). 1148 bytes result sent to driver
17:38:07.250 INFO TaskSetManager - Finished task 0.0 in stage 154.0 (TID 210) in 33 ms on localhost (executor driver) (1/1)
17:38:07.250 INFO TaskSchedulerImpl - Removed TaskSet 154.0, whose tasks have all completed, from pool
17:38:07.250 INFO DAGScheduler - ShuffleMapStage 154 (mapToPair at SparkUtils.java:161) finished in 0.045 s
17:38:07.250 INFO DAGScheduler - looking for newly runnable stages
17:38:07.250 INFO DAGScheduler - running: HashSet()
17:38:07.250 INFO DAGScheduler - waiting: HashSet(ResultStage 155)
17:38:07.250 INFO DAGScheduler - failed: HashSet()
17:38:07.250 INFO DAGScheduler - Submitting ResultStage 155 (MapPartitionsRDD[723] at mapToPair at CramSink.java:89), which has no missing parents
17:38:07.256 INFO MemoryStore - Block broadcast_297 stored as values in memory (estimated size 153.2 KiB, free 1919.1 MiB)
17:38:07.256 INFO MemoryStore - Block broadcast_297_piece0 stored as bytes in memory (estimated size 58.1 KiB, free 1919.0 MiB)
17:38:07.257 INFO BlockManagerInfo - Added broadcast_297_piece0 in memory on localhost:37953 (size: 58.1 KiB, free: 1919.8 MiB)
17:38:07.257 INFO SparkContext - Created broadcast 297 from broadcast at DAGScheduler.scala:1580
17:38:07.257 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 155 (MapPartitionsRDD[723] at mapToPair at CramSink.java:89) (first 15 tasks are for partitions Vector(0))
17:38:07.257 INFO TaskSchedulerImpl - Adding task set 155.0 with 1 tasks resource profile 0
17:38:07.257 INFO TaskSetManager - Starting task 0.0 in stage 155.0 (TID 211) (localhost, executor driver, partition 0, NODE_LOCAL, 7513 bytes)
17:38:07.258 INFO Executor - Running task 0.0 in stage 155.0 (TID 211)
17:38:07.262 INFO ShuffleBlockFetcherIterator - Getting 1 (82.3 KiB) non-empty blocks including 1 (82.3 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
17:38:07.262 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
17:38:07.269 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:07.269 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:07.269 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:07.269 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:07.269 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:07.269 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:07.308 INFO FileOutputCommitter - Saved output of task 'attempt_202510271738077656795335930511340_0723_r_000000_0' to file:/tmp/ReadsSparkSinkUnitTest52024556876441371598.cram.parts/_temporary/0/task_202510271738077656795335930511340_0723_r_000000
17:38:07.309 INFO SparkHadoopMapRedUtil - attempt_202510271738077656795335930511340_0723_r_000000_0: Committed. Elapsed time: 0 ms.
17:38:07.309 INFO Executor - Finished task 0.0 in stage 155.0 (TID 211). 1858 bytes result sent to driver
17:38:07.309 INFO TaskSetManager - Finished task 0.0 in stage 155.0 (TID 211) in 52 ms on localhost (executor driver) (1/1)
17:38:07.309 INFO TaskSchedulerImpl - Removed TaskSet 155.0, whose tasks have all completed, from pool
17:38:07.310 INFO DAGScheduler - ResultStage 155 (runJob at SparkHadoopWriter.scala:83) finished in 0.060 s
17:38:07.310 INFO DAGScheduler - Job 111 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:07.310 INFO TaskSchedulerImpl - Killing all running tasks in stage 155: Stage finished
17:38:07.310 INFO DAGScheduler - Job 111 finished: runJob at SparkHadoopWriter.scala:83, took 0.105401 s
17:38:07.310 INFO SparkHadoopWriter - Start to commit write Job job_202510271738077656795335930511340_0723.
17:38:07.313 INFO SparkHadoopWriter - Write Job job_202510271738077656795335930511340_0723 committed. Elapsed time: 3 ms.
17:38:07.322 INFO HadoopFileSystemWrapper - Concatenating 3 parts to /tmp/ReadsSparkSinkUnitTest52024556876441371598.cram
17:38:07.325 INFO HadoopFileSystemWrapper - Concatenating to /tmp/ReadsSparkSinkUnitTest52024556876441371598.cram done
17:38:07.327 INFO MemoryStore - Block broadcast_298 stored as values in memory (estimated size 504.0 B, free 1919.0 MiB)
17:38:07.327 INFO MemoryStore - Block broadcast_298_piece0 stored as bytes in memory (estimated size 159.0 B, free 1919.0 MiB)
17:38:07.327 INFO BlockManagerInfo - Added broadcast_298_piece0 in memory on localhost:37953 (size: 159.0 B, free: 1919.8 MiB)
17:38:07.327 INFO SparkContext - Created broadcast 298 from broadcast at CramSource.java:114
17:38:07.328 INFO MemoryStore - Block broadcast_299 stored as values in memory (estimated size 297.9 KiB, free 1918.8 MiB)
17:38:07.334 INFO MemoryStore - Block broadcast_299_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1918.7 MiB)
17:38:07.334 INFO BlockManagerInfo - Added broadcast_299_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.7 MiB)
17:38:07.334 INFO SparkContext - Created broadcast 299 from newAPIHadoopFile at PathSplitSource.java:96
17:38:07.348 INFO FileInputFormat - Total input files to process : 1
17:38:07.370 INFO SparkContext - Starting job: collect at ReadsSparkSinkUnitTest.java:182
17:38:07.370 INFO DAGScheduler - Got job 112 (collect at ReadsSparkSinkUnitTest.java:182) with 1 output partitions
17:38:07.370 INFO DAGScheduler - Final stage: ResultStage 156 (collect at ReadsSparkSinkUnitTest.java:182)
17:38:07.370 INFO DAGScheduler - Parents of final stage: List()
17:38:07.370 INFO DAGScheduler - Missing parents: List()
17:38:07.370 INFO DAGScheduler - Submitting ResultStage 156 (MapPartitionsRDD[729] at filter at ReadsSparkSource.java:96), which has no missing parents
17:38:07.381 INFO MemoryStore - Block broadcast_300 stored as values in memory (estimated size 286.8 KiB, free 1918.4 MiB)
17:38:07.384 INFO BlockManagerInfo - Removed broadcast_296_piece0 on localhost:37953 in memory (size: 107.3 KiB, free: 1919.8 MiB)
17:38:07.384 INFO MemoryStore - Block broadcast_300_piece0 stored as bytes in memory (estimated size 103.6 KiB, free 1918.7 MiB)
17:38:07.385 INFO BlockManagerInfo - Added broadcast_300_piece0 in memory on localhost:37953 (size: 103.6 KiB, free: 1919.7 MiB)
17:38:07.385 INFO SparkContext - Created broadcast 300 from broadcast at DAGScheduler.scala:1580
17:38:07.385 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 156 (MapPartitionsRDD[729] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:38:07.385 INFO TaskSchedulerImpl - Adding task set 156.0 with 1 tasks resource profile 0
17:38:07.385 INFO BlockManagerInfo - Removed broadcast_294_piece0 on localhost:37953 in memory (size: 1473.0 B, free: 1919.7 MiB)
17:38:07.385 INFO TaskSetManager - Starting task 0.0 in stage 156.0 (TID 212) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7810 bytes)
17:38:07.386 INFO Executor - Running task 0.0 in stage 156.0 (TID 212)
17:38:07.386 INFO BlockManagerInfo - Removed broadcast_297_piece0 on localhost:37953 in memory (size: 58.1 KiB, free: 1919.8 MiB)
17:38:07.388 INFO BlockManagerInfo - Removed broadcast_295_piece0 on localhost:37953 in memory (size: 1473.0 B, free: 1919.8 MiB)
17:38:07.407 INFO NewHadoopRDD - Input split: file:/tmp/ReadsSparkSinkUnitTest52024556876441371598.cram:0+43713
17:38:07.423 INFO Executor - Finished task 0.0 in stage 156.0 (TID 212). 154101 bytes result sent to driver
17:38:07.424 INFO TaskSetManager - Finished task 0.0 in stage 156.0 (TID 212) in 39 ms on localhost (executor driver) (1/1)
17:38:07.424 INFO TaskSchedulerImpl - Removed TaskSet 156.0, whose tasks have all completed, from pool
17:38:07.424 INFO DAGScheduler - ResultStage 156 (collect at ReadsSparkSinkUnitTest.java:182) finished in 0.053 s
17:38:07.424 INFO DAGScheduler - Job 112 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:07.424 INFO TaskSchedulerImpl - Killing all running tasks in stage 156: Stage finished
17:38:07.424 INFO DAGScheduler - Job 112 finished: collect at ReadsSparkSinkUnitTest.java:182, took 0.054293 s
17:38:07.429 INFO SparkContext - Starting job: count at ReadsSparkSinkUnitTest.java:185
17:38:07.429 INFO DAGScheduler - Got job 113 (count at ReadsSparkSinkUnitTest.java:185) with 1 output partitions
17:38:07.429 INFO DAGScheduler - Final stage: ResultStage 157 (count at ReadsSparkSinkUnitTest.java:185)
17:38:07.429 INFO DAGScheduler - Parents of final stage: List()
17:38:07.429 INFO DAGScheduler - Missing parents: List()
17:38:07.430 INFO DAGScheduler - Submitting ResultStage 157 (MapPartitionsRDD[712] at filter at ReadsSparkSource.java:96), which has no missing parents
17:38:07.440 INFO MemoryStore - Block broadcast_301 stored as values in memory (estimated size 286.8 KiB, free 1918.7 MiB)
17:38:07.441 INFO MemoryStore - Block broadcast_301_piece0 stored as bytes in memory (estimated size 103.6 KiB, free 1918.6 MiB)
17:38:07.441 INFO BlockManagerInfo - Added broadcast_301_piece0 in memory on localhost:37953 (size: 103.6 KiB, free: 1919.7 MiB)
17:38:07.441 INFO SparkContext - Created broadcast 301 from broadcast at DAGScheduler.scala:1580
17:38:07.441 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 157 (MapPartitionsRDD[712] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:38:07.441 INFO TaskSchedulerImpl - Adding task set 157.0 with 1 tasks resource profile 0
17:38:07.442 INFO TaskSetManager - Starting task 0.0 in stage 157.0 (TID 213) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7880 bytes)
17:38:07.442 INFO Executor - Running task 0.0 in stage 157.0 (TID 213)
17:38:07.463 INFO NewHadoopRDD - Input split: file:/home/runner/work/gatk/gatk/src/test/resources/org/broadinstitute/hellbender/tools/BQSR/NA12878.chr17_69k_70k.dictFix.cram:0+50619
17:38:07.469 INFO Executor - Finished task 0.0 in stage 157.0 (TID 213). 989 bytes result sent to driver
17:38:07.470 INFO TaskSetManager - Finished task 0.0 in stage 157.0 (TID 213) in 28 ms on localhost (executor driver) (1/1)
17:38:07.470 INFO TaskSchedulerImpl - Removed TaskSet 157.0, whose tasks have all completed, from pool
17:38:07.470 INFO DAGScheduler - ResultStage 157 (count at ReadsSparkSinkUnitTest.java:185) finished in 0.040 s
17:38:07.470 INFO DAGScheduler - Job 113 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:07.470 INFO TaskSchedulerImpl - Killing all running tasks in stage 157: Stage finished
17:38:07.470 INFO DAGScheduler - Job 113 finished: count at ReadsSparkSinkUnitTest.java:185, took 0.040946 s
17:38:07.473 INFO SparkContext - Starting job: count at ReadsSparkSinkUnitTest.java:185
17:38:07.473 INFO DAGScheduler - Got job 114 (count at ReadsSparkSinkUnitTest.java:185) with 1 output partitions
17:38:07.473 INFO DAGScheduler - Final stage: ResultStage 158 (count at ReadsSparkSinkUnitTest.java:185)
17:38:07.473 INFO DAGScheduler - Parents of final stage: List()
17:38:07.473 INFO DAGScheduler - Missing parents: List()
17:38:07.474 INFO DAGScheduler - Submitting ResultStage 158 (MapPartitionsRDD[729] at filter at ReadsSparkSource.java:96), which has no missing parents
17:38:07.484 INFO MemoryStore - Block broadcast_302 stored as values in memory (estimated size 286.8 KiB, free 1918.3 MiB)
17:38:07.485 INFO MemoryStore - Block broadcast_302_piece0 stored as bytes in memory (estimated size 103.6 KiB, free 1918.2 MiB)
17:38:07.485 INFO BlockManagerInfo - Added broadcast_302_piece0 in memory on localhost:37953 (size: 103.6 KiB, free: 1919.6 MiB)
17:38:07.485 INFO SparkContext - Created broadcast 302 from broadcast at DAGScheduler.scala:1580
17:38:07.485 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 158 (MapPartitionsRDD[729] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:38:07.485 INFO TaskSchedulerImpl - Adding task set 158.0 with 1 tasks resource profile 0
17:38:07.486 INFO TaskSetManager - Starting task 0.0 in stage 158.0 (TID 214) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7810 bytes)
17:38:07.486 INFO Executor - Running task 0.0 in stage 158.0 (TID 214)
17:38:07.507 INFO NewHadoopRDD - Input split: file:/tmp/ReadsSparkSinkUnitTest52024556876441371598.cram:0+43713
17:38:07.517 INFO Executor - Finished task 0.0 in stage 158.0 (TID 214). 989 bytes result sent to driver
17:38:07.517 INFO TaskSetManager - Finished task 0.0 in stage 158.0 (TID 214) in 31 ms on localhost (executor driver) (1/1)
17:38:07.517 INFO TaskSchedulerImpl - Removed TaskSet 158.0, whose tasks have all completed, from pool
17:38:07.517 INFO DAGScheduler - ResultStage 158 (count at ReadsSparkSinkUnitTest.java:185) finished in 0.043 s
17:38:07.517 INFO DAGScheduler - Job 114 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:07.517 INFO TaskSchedulerImpl - Killing all running tasks in stage 158: Stage finished
17:38:07.517 INFO DAGScheduler - Job 114 finished: count at ReadsSparkSinkUnitTest.java:185, took 0.044257 s
17:38:07.518 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:07.520 INFO MemoryStore - Block broadcast_303 stored as values in memory (estimated size 297.9 KiB, free 1917.9 MiB)
17:38:07.526 INFO MemoryStore - Block broadcast_303_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1917.8 MiB)
17:38:07.526 INFO BlockManagerInfo - Added broadcast_303_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.5 MiB)
17:38:07.526 INFO SparkContext - Created broadcast 303 from newAPIHadoopFile at PathSplitSource.java:96
17:38:07.545 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:07.547 INFO MemoryStore - Block broadcast_304 stored as values in memory (estimated size 297.9 KiB, free 1917.5 MiB)
17:38:07.553 INFO MemoryStore - Block broadcast_304_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1917.5 MiB)
17:38:07.553 INFO BlockManagerInfo - Added broadcast_304_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.5 MiB)
17:38:07.553 INFO SparkContext - Created broadcast 304 from newAPIHadoopFile at PathSplitSource.java:96
17:38:07.572 INFO FileInputFormat - Total input files to process : 1
17:38:07.574 INFO MemoryStore - Block broadcast_305 stored as values in memory (estimated size 160.7 KiB, free 1917.3 MiB)
17:38:07.575 INFO MemoryStore - Block broadcast_305_piece0 stored as bytes in memory (estimated size 9.6 KiB, free 1917.3 MiB)
17:38:07.575 INFO BlockManagerInfo - Added broadcast_305_piece0 in memory on localhost:37953 (size: 9.6 KiB, free: 1919.5 MiB)
17:38:07.575 INFO SparkContext - Created broadcast 305 from broadcast at ReadsSparkSink.java:133
17:38:07.579 INFO HadoopMapRedCommitProtocol - Using output committer class org.apache.hadoop.mapred.FileOutputCommitter
17:38:07.579 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:07.579 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:07.592 INFO SparkContext - Starting job: runJob at SparkHadoopWriter.scala:83
17:38:07.593 INFO DAGScheduler - Registering RDD 743 (mapToPair at SparkUtils.java:161) as input to shuffle 33
17:38:07.593 INFO DAGScheduler - Got job 115 (runJob at SparkHadoopWriter.scala:83) with 1 output partitions
17:38:07.593 INFO DAGScheduler - Final stage: ResultStage 160 (runJob at SparkHadoopWriter.scala:83)
17:38:07.593 INFO DAGScheduler - Parents of final stage: List(ShuffleMapStage 159)
17:38:07.593 INFO DAGScheduler - Missing parents: List(ShuffleMapStage 159)
17:38:07.593 INFO DAGScheduler - Submitting ShuffleMapStage 159 (MapPartitionsRDD[743] at mapToPair at SparkUtils.java:161), which has no missing parents
17:38:07.609 INFO MemoryStore - Block broadcast_306 stored as values in memory (estimated size 520.4 KiB, free 1916.8 MiB)
17:38:07.610 INFO MemoryStore - Block broadcast_306_piece0 stored as bytes in memory (estimated size 166.1 KiB, free 1916.7 MiB)
17:38:07.610 INFO BlockManagerInfo - Added broadcast_306_piece0 in memory on localhost:37953 (size: 166.1 KiB, free: 1919.3 MiB)
17:38:07.610 INFO SparkContext - Created broadcast 306 from broadcast at DAGScheduler.scala:1580
17:38:07.610 INFO DAGScheduler - Submitting 1 missing tasks from ShuffleMapStage 159 (MapPartitionsRDD[743] at mapToPair at SparkUtils.java:161) (first 15 tasks are for partitions Vector(0))
17:38:07.610 INFO TaskSchedulerImpl - Adding task set 159.0 with 1 tasks resource profile 0
17:38:07.611 INFO TaskSetManager - Starting task 0.0 in stage 159.0 (TID 215) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7861 bytes)
17:38:07.611 INFO Executor - Running task 0.0 in stage 159.0 (TID 215)
17:38:07.643 INFO NewHadoopRDD - Input split: file:/home/runner/work/gatk/gatk/src/test/resources/org/broadinstitute/hellbender/tools/BQSR/HiSeq.1mb.1RG.2k_lines.bam:0+222075
17:38:07.643 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:07.644 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:07.651 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:07.664 INFO Executor - Finished task 0.0 in stage 159.0 (TID 215). 1234 bytes result sent to driver
17:38:07.664 INFO BlockManagerInfo - Removed broadcast_300_piece0 on localhost:37953 in memory (size: 103.6 KiB, free: 1919.4 MiB)
17:38:07.664 INFO TaskSetManager - Finished task 0.0 in stage 159.0 (TID 215) in 53 ms on localhost (executor driver) (1/1)
17:38:07.664 INFO TaskSchedulerImpl - Removed TaskSet 159.0, whose tasks have all completed, from pool
17:38:07.664 INFO DAGScheduler - ShuffleMapStage 159 (mapToPair at SparkUtils.java:161) finished in 0.071 s
17:38:07.664 INFO DAGScheduler - looking for newly runnable stages
17:38:07.664 INFO DAGScheduler - running: HashSet()
17:38:07.664 INFO DAGScheduler - waiting: HashSet(ResultStage 160)
17:38:07.664 INFO DAGScheduler - failed: HashSet()
17:38:07.665 INFO DAGScheduler - Submitting ResultStage 160 (MapPartitionsRDD[749] at saveAsTextFile at SamSink.java:65), which has no missing parents
17:38:07.665 INFO BlockManagerInfo - Removed broadcast_301_piece0 on localhost:37953 in memory (size: 103.6 KiB, free: 1919.5 MiB)
17:38:07.666 INFO BlockManagerInfo - Removed broadcast_291_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.6 MiB)
17:38:07.666 INFO BlockManagerInfo - Removed broadcast_304_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.6 MiB)
17:38:07.666 INFO BlockManagerInfo - Removed broadcast_290_piece0 on localhost:37953 in memory (size: 228.0 B, free: 1919.6 MiB)
17:38:07.669 INFO BlockManagerInfo - Removed broadcast_298_piece0 on localhost:37953 in memory (size: 159.0 B, free: 1919.6 MiB)
17:38:07.669 INFO BlockManagerInfo - Removed broadcast_299_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.7 MiB)
17:38:07.670 INFO BlockManagerInfo - Removed broadcast_302_piece0 on localhost:37953 in memory (size: 103.6 KiB, free: 1919.8 MiB)
17:38:07.671 INFO MemoryStore - Block broadcast_307 stored as values in memory (estimated size 241.1 KiB, free 1918.6 MiB)
17:38:07.671 INFO MemoryStore - Block broadcast_307_piece0 stored as bytes in memory (estimated size 66.9 KiB, free 1918.5 MiB)
17:38:07.671 INFO BlockManagerInfo - Added broadcast_307_piece0 in memory on localhost:37953 (size: 66.9 KiB, free: 1919.7 MiB)
17:38:07.672 INFO SparkContext - Created broadcast 307 from broadcast at DAGScheduler.scala:1580
17:38:07.672 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 160 (MapPartitionsRDD[749] at saveAsTextFile at SamSink.java:65) (first 15 tasks are for partitions Vector(0))
17:38:07.672 INFO TaskSchedulerImpl - Adding task set 160.0 with 1 tasks resource profile 0
17:38:07.672 INFO TaskSetManager - Starting task 0.0 in stage 160.0 (TID 216) (localhost, executor driver, partition 0, NODE_LOCAL, 7513 bytes)
17:38:07.673 INFO Executor - Running task 0.0 in stage 160.0 (TID 216)
17:38:07.678 INFO ShuffleBlockFetcherIterator - Getting 1 (343.8 KiB) non-empty blocks including 1 (343.8 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
17:38:07.678 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
17:38:07.688 INFO HadoopMapRedCommitProtocol - Using output committer class org.apache.hadoop.mapred.FileOutputCommitter
17:38:07.688 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:07.688 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:07.703 INFO FileOutputCommitter - Saved output of task 'attempt_202510271738072064204901244168178_0749_m_000000_0' to file:/tmp/ReadsSparkSinkUnitTest616383566067458748146.sam.parts/_temporary/0/task_202510271738072064204901244168178_0749_m_000000
17:38:07.703 INFO SparkHadoopMapRedUtil - attempt_202510271738072064204901244168178_0749_m_000000_0: Committed. Elapsed time: 0 ms.
17:38:07.703 INFO Executor - Finished task 0.0 in stage 160.0 (TID 216). 1858 bytes result sent to driver
17:38:07.704 INFO TaskSetManager - Finished task 0.0 in stage 160.0 (TID 216) in 32 ms on localhost (executor driver) (1/1)
17:38:07.704 INFO TaskSchedulerImpl - Removed TaskSet 160.0, whose tasks have all completed, from pool
17:38:07.704 INFO DAGScheduler - ResultStage 160 (runJob at SparkHadoopWriter.scala:83) finished in 0.039 s
17:38:07.704 INFO DAGScheduler - Job 115 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:07.704 INFO TaskSchedulerImpl - Killing all running tasks in stage 160: Stage finished
17:38:07.704 INFO DAGScheduler - Job 115 finished: runJob at SparkHadoopWriter.scala:83, took 0.111656 s
17:38:07.704 INFO SparkHadoopWriter - Start to commit write Job job_202510271738072064204901244168178_0749.
17:38:07.708 INFO SparkHadoopWriter - Write Job job_202510271738072064204901244168178_0749 committed. Elapsed time: 3 ms.
17:38:07.713 INFO HadoopFileSystemWrapper - Concatenating 2 parts to /tmp/ReadsSparkSinkUnitTest616383566067458748146.sam
17:38:07.716 INFO HadoopFileSystemWrapper - Concatenating to /tmp/ReadsSparkSinkUnitTest616383566067458748146.sam done
WARNING 2025-10-27 17:38:07 SamReaderFactory Unable to detect file format from input URL or stream, assuming SAM format.
WARNING 2025-10-27 17:38:07 SamReaderFactory Unable to detect file format from input URL or stream, assuming SAM format.
17:38:07.720 INFO MemoryStore - Block broadcast_308 stored as values in memory (estimated size 160.7 KiB, free 1918.4 MiB)
17:38:07.720 INFO MemoryStore - Block broadcast_308_piece0 stored as bytes in memory (estimated size 9.6 KiB, free 1918.4 MiB)
17:38:07.720 INFO BlockManagerInfo - Added broadcast_308_piece0 in memory on localhost:37953 (size: 9.6 KiB, free: 1919.7 MiB)
17:38:07.721 INFO SparkContext - Created broadcast 308 from broadcast at SamSource.java:78
17:38:07.722 INFO MemoryStore - Block broadcast_309 stored as values in memory (estimated size 297.9 KiB, free 1918.1 MiB)
17:38:07.727 INFO MemoryStore - Block broadcast_309_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1918.0 MiB)
17:38:07.727 INFO BlockManagerInfo - Added broadcast_309_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.7 MiB)
17:38:07.728 INFO SparkContext - Created broadcast 309 from newAPIHadoopFile at SamSource.java:108
17:38:07.730 INFO FileInputFormat - Total input files to process : 1
17:38:07.734 INFO SparkContext - Starting job: collect at ReadsSparkSinkUnitTest.java:182
17:38:07.734 INFO DAGScheduler - Got job 116 (collect at ReadsSparkSinkUnitTest.java:182) with 1 output partitions
17:38:07.734 INFO DAGScheduler - Final stage: ResultStage 161 (collect at ReadsSparkSinkUnitTest.java:182)
17:38:07.734 INFO DAGScheduler - Parents of final stage: List()
17:38:07.734 INFO DAGScheduler - Missing parents: List()
17:38:07.734 INFO DAGScheduler - Submitting ResultStage 161 (MapPartitionsRDD[754] at filter at ReadsSparkSource.java:96), which has no missing parents
17:38:07.735 INFO MemoryStore - Block broadcast_310 stored as values in memory (estimated size 7.5 KiB, free 1918.0 MiB)
17:38:07.735 INFO MemoryStore - Block broadcast_310_piece0 stored as bytes in memory (estimated size 3.8 KiB, free 1918.0 MiB)
17:38:07.735 INFO BlockManagerInfo - Added broadcast_310_piece0 in memory on localhost:37953 (size: 3.8 KiB, free: 1919.7 MiB)
17:38:07.736 INFO SparkContext - Created broadcast 310 from broadcast at DAGScheduler.scala:1580
17:38:07.736 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 161 (MapPartitionsRDD[754] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:38:07.736 INFO TaskSchedulerImpl - Adding task set 161.0 with 1 tasks resource profile 0
17:38:07.736 INFO TaskSetManager - Starting task 0.0 in stage 161.0 (TID 217) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7810 bytes)
17:38:07.737 INFO Executor - Running task 0.0 in stage 161.0 (TID 217)
17:38:07.738 INFO NewHadoopRDD - Input split: file:/tmp/ReadsSparkSinkUnitTest616383566067458748146.sam:0+847558
17:38:07.749 INFO Executor - Finished task 0.0 in stage 161.0 (TID 217). 651483 bytes result sent to driver
17:38:07.750 INFO TaskSetManager - Finished task 0.0 in stage 161.0 (TID 217) in 14 ms on localhost (executor driver) (1/1)
17:38:07.750 INFO TaskSchedulerImpl - Removed TaskSet 161.0, whose tasks have all completed, from pool
17:38:07.750 INFO DAGScheduler - ResultStage 161 (collect at ReadsSparkSinkUnitTest.java:182) finished in 0.015 s
17:38:07.750 INFO DAGScheduler - Job 116 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:07.751 INFO TaskSchedulerImpl - Killing all running tasks in stage 161: Stage finished
17:38:07.751 INFO DAGScheduler - Job 116 finished: collect at ReadsSparkSinkUnitTest.java:182, took 0.016578 s
17:38:07.760 INFO SparkContext - Starting job: count at ReadsSparkSinkUnitTest.java:185
17:38:07.761 INFO DAGScheduler - Got job 117 (count at ReadsSparkSinkUnitTest.java:185) with 1 output partitions
17:38:07.761 INFO DAGScheduler - Final stage: ResultStage 162 (count at ReadsSparkSinkUnitTest.java:185)
17:38:07.761 INFO DAGScheduler - Parents of final stage: List()
17:38:07.761 INFO DAGScheduler - Missing parents: List()
17:38:07.761 INFO DAGScheduler - Submitting ResultStage 162 (MapPartitionsRDD[736] at filter at ReadsSparkSource.java:96), which has no missing parents
17:38:07.776 INFO MemoryStore - Block broadcast_311 stored as values in memory (estimated size 426.1 KiB, free 1917.6 MiB)
17:38:07.777 INFO MemoryStore - Block broadcast_311_piece0 stored as bytes in memory (estimated size 153.6 KiB, free 1917.4 MiB)
17:38:07.777 INFO BlockManagerInfo - Added broadcast_311_piece0 in memory on localhost:37953 (size: 153.6 KiB, free: 1919.5 MiB)
17:38:07.778 INFO SparkContext - Created broadcast 311 from broadcast at DAGScheduler.scala:1580
17:38:07.778 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 162 (MapPartitionsRDD[736] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:38:07.778 INFO TaskSchedulerImpl - Adding task set 162.0 with 1 tasks resource profile 0
17:38:07.778 INFO TaskSetManager - Starting task 0.0 in stage 162.0 (TID 218) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7872 bytes)
17:38:07.779 INFO Executor - Running task 0.0 in stage 162.0 (TID 218)
17:38:07.810 INFO NewHadoopRDD - Input split: file:/home/runner/work/gatk/gatk/src/test/resources/org/broadinstitute/hellbender/tools/BQSR/HiSeq.1mb.1RG.2k_lines.bam:0+222075
17:38:07.811 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:07.812 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:07.819 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:07.823 INFO Executor - Finished task 0.0 in stage 162.0 (TID 218). 989 bytes result sent to driver
17:38:07.823 INFO TaskSetManager - Finished task 0.0 in stage 162.0 (TID 218) in 45 ms on localhost (executor driver) (1/1)
17:38:07.823 INFO TaskSchedulerImpl - Removed TaskSet 162.0, whose tasks have all completed, from pool
17:38:07.823 INFO DAGScheduler - ResultStage 162 (count at ReadsSparkSinkUnitTest.java:185) finished in 0.062 s
17:38:07.823 INFO DAGScheduler - Job 117 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:07.823 INFO TaskSchedulerImpl - Killing all running tasks in stage 162: Stage finished
17:38:07.823 INFO DAGScheduler - Job 117 finished: count at ReadsSparkSinkUnitTest.java:185, took 0.063148 s
17:38:07.827 INFO SparkContext - Starting job: count at ReadsSparkSinkUnitTest.java:185
17:38:07.827 INFO DAGScheduler - Got job 118 (count at ReadsSparkSinkUnitTest.java:185) with 1 output partitions
17:38:07.827 INFO DAGScheduler - Final stage: ResultStage 163 (count at ReadsSparkSinkUnitTest.java:185)
17:38:07.827 INFO DAGScheduler - Parents of final stage: List()
17:38:07.827 INFO DAGScheduler - Missing parents: List()
17:38:07.827 INFO DAGScheduler - Submitting ResultStage 163 (MapPartitionsRDD[754] at filter at ReadsSparkSource.java:96), which has no missing parents
17:38:07.828 INFO MemoryStore - Block broadcast_312 stored as values in memory (estimated size 7.4 KiB, free 1917.4 MiB)
17:38:07.828 INFO MemoryStore - Block broadcast_312_piece0 stored as bytes in memory (estimated size 3.8 KiB, free 1917.4 MiB)
17:38:07.828 INFO BlockManagerInfo - Added broadcast_312_piece0 in memory on localhost:37953 (size: 3.8 KiB, free: 1919.5 MiB)
17:38:07.828 INFO SparkContext - Created broadcast 312 from broadcast at DAGScheduler.scala:1580
17:38:07.828 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 163 (MapPartitionsRDD[754] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:38:07.828 INFO TaskSchedulerImpl - Adding task set 163.0 with 1 tasks resource profile 0
17:38:07.829 INFO TaskSetManager - Starting task 0.0 in stage 163.0 (TID 219) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7810 bytes)
17:38:07.829 INFO Executor - Running task 0.0 in stage 163.0 (TID 219)
17:38:07.830 INFO NewHadoopRDD - Input split: file:/tmp/ReadsSparkSinkUnitTest616383566067458748146.sam:0+847558
17:38:07.837 INFO Executor - Finished task 0.0 in stage 163.0 (TID 219). 946 bytes result sent to driver
17:38:07.837 INFO TaskSetManager - Finished task 0.0 in stage 163.0 (TID 219) in 8 ms on localhost (executor driver) (1/1)
17:38:07.837 INFO TaskSchedulerImpl - Removed TaskSet 163.0, whose tasks have all completed, from pool
17:38:07.837 INFO DAGScheduler - ResultStage 163 (count at ReadsSparkSinkUnitTest.java:185) finished in 0.010 s
17:38:07.837 INFO DAGScheduler - Job 118 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:07.837 INFO TaskSchedulerImpl - Killing all running tasks in stage 163: Stage finished
17:38:07.837 INFO DAGScheduler - Job 118 finished: count at ReadsSparkSinkUnitTest.java:185, took 0.010903 s
WARNING 2025-10-27 17:38:07 SamReaderFactory Unable to detect file format from input URL or stream, assuming SAM format.
WARNING 2025-10-27 17:38:07 SamReaderFactory Unable to detect file format from input URL or stream, assuming SAM format.
17:38:07.840 INFO MemoryStore - Block broadcast_313 stored as values in memory (estimated size 21.0 KiB, free 1917.4 MiB)
17:38:07.844 INFO MemoryStore - Block broadcast_313_piece0 stored as bytes in memory (estimated size 2.4 KiB, free 1917.4 MiB)
17:38:07.844 INFO BlockManagerInfo - Added broadcast_313_piece0 in memory on localhost:37953 (size: 2.4 KiB, free: 1919.5 MiB)
17:38:07.844 INFO BlockManagerInfo - Removed broadcast_309_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.5 MiB)
17:38:07.845 INFO SparkContext - Created broadcast 313 from broadcast at SamSource.java:78
17:38:07.845 INFO BlockManagerInfo - Removed broadcast_310_piece0 on localhost:37953 in memory (size: 3.8 KiB, free: 1919.5 MiB)
17:38:07.845 INFO BlockManagerInfo - Removed broadcast_312_piece0 on localhost:37953 in memory (size: 3.8 KiB, free: 1919.6 MiB)
17:38:07.846 INFO MemoryStore - Block broadcast_314 stored as values in memory (estimated size 298.0 KiB, free 1917.5 MiB)
17:38:07.846 INFO BlockManagerInfo - Removed broadcast_311_piece0 on localhost:37953 in memory (size: 153.6 KiB, free: 1919.7 MiB)
17:38:07.848 INFO BlockManagerInfo - Removed broadcast_308_piece0 on localhost:37953 in memory (size: 9.6 KiB, free: 1919.7 MiB)
17:38:07.848 INFO BlockManagerInfo - Removed broadcast_303_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.8 MiB)
17:38:07.849 INFO BlockManagerInfo - Removed broadcast_307_piece0 on localhost:37953 in memory (size: 66.9 KiB, free: 1919.8 MiB)
17:38:07.849 INFO BlockManagerInfo - Removed broadcast_306_piece0 on localhost:37953 in memory (size: 166.1 KiB, free: 1920.0 MiB)
17:38:07.850 INFO BlockManagerInfo - Removed broadcast_305_piece0 on localhost:37953 in memory (size: 9.6 KiB, free: 1920.0 MiB)
17:38:07.851 INFO MemoryStore - Block broadcast_314_piece0 stored as bytes in memory (estimated size 50.3 KiB, free 1919.6 MiB)
17:38:07.851 INFO BlockManagerInfo - Added broadcast_314_piece0 in memory on localhost:37953 (size: 50.3 KiB, free: 1919.9 MiB)
17:38:07.852 INFO SparkContext - Created broadcast 314 from newAPIHadoopFile at SamSource.java:108
17:38:07.856 INFO FileInputFormat - Total input files to process : 1
17:38:07.859 INFO SparkContext - Starting job: collect at SparkUtils.java:205
17:38:07.859 INFO DAGScheduler - Got job 119 (collect at SparkUtils.java:205) with 1 output partitions
17:38:07.859 INFO DAGScheduler - Final stage: ResultStage 164 (collect at SparkUtils.java:205)
17:38:07.859 INFO DAGScheduler - Parents of final stage: List()
17:38:07.859 INFO DAGScheduler - Missing parents: List()
17:38:07.859 INFO DAGScheduler - Submitting ResultStage 164 (MapPartitionsRDD[760] at mapPartitions at SparkUtils.java:188), which has no missing parents
17:38:07.860 INFO MemoryStore - Block broadcast_315 stored as values in memory (estimated size 7.9 KiB, free 1919.6 MiB)
17:38:07.860 INFO MemoryStore - Block broadcast_315_piece0 stored as bytes in memory (estimated size 3.9 KiB, free 1919.6 MiB)
17:38:07.860 INFO BlockManagerInfo - Added broadcast_315_piece0 in memory on localhost:37953 (size: 3.9 KiB, free: 1919.9 MiB)
17:38:07.861 INFO SparkContext - Created broadcast 315 from broadcast at DAGScheduler.scala:1580
17:38:07.861 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 164 (MapPartitionsRDD[760] at mapPartitions at SparkUtils.java:188) (first 15 tasks are for partitions Vector(0))
17:38:07.861 INFO TaskSchedulerImpl - Adding task set 164.0 with 1 tasks resource profile 0
17:38:07.861 INFO TaskSetManager - Starting task 0.0 in stage 164.0 (TID 220) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7936 bytes)
17:38:07.861 INFO Executor - Running task 0.0 in stage 164.0 (TID 220)
17:38:07.863 INFO NewHadoopRDD - Input split: file:/home/runner/work/gatk/gatk/src/test/resources/org/broadinstitute/hellbender/engine/CEUTrio.HiSeq.WGS.b37.NA12878.20.21.10000000-10000020.with.unmapped.queryname.samtools.sam:0+224884
17:38:07.865 INFO Executor - Finished task 0.0 in stage 164.0 (TID 220). 1700 bytes result sent to driver
17:38:07.866 INFO TaskSetManager - Finished task 0.0 in stage 164.0 (TID 220) in 5 ms on localhost (executor driver) (1/1)
17:38:07.866 INFO TaskSchedulerImpl - Removed TaskSet 164.0, whose tasks have all completed, from pool
17:38:07.866 INFO DAGScheduler - ResultStage 164 (collect at SparkUtils.java:205) finished in 0.006 s
17:38:07.866 INFO DAGScheduler - Job 119 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:07.866 INFO TaskSchedulerImpl - Killing all running tasks in stage 164: Stage finished
17:38:07.867 INFO DAGScheduler - Job 119 finished: collect at SparkUtils.java:205, took 0.007498 s
WARNING 2025-10-27 17:38:07 SamReaderFactory Unable to detect file format from input URL or stream, assuming SAM format.
WARNING 2025-10-27 17:38:07 SamReaderFactory Unable to detect file format from input URL or stream, assuming SAM format.
17:38:07.871 INFO MemoryStore - Block broadcast_316 stored as values in memory (estimated size 21.0 KiB, free 1919.6 MiB)
17:38:07.871 INFO MemoryStore - Block broadcast_316_piece0 stored as bytes in memory (estimated size 2.4 KiB, free 1919.6 MiB)
17:38:07.872 INFO BlockManagerInfo - Added broadcast_316_piece0 in memory on localhost:37953 (size: 2.4 KiB, free: 1919.9 MiB)
17:38:07.872 INFO SparkContext - Created broadcast 316 from broadcast at SamSource.java:78
17:38:07.873 INFO MemoryStore - Block broadcast_317 stored as values in memory (estimated size 298.0 KiB, free 1919.3 MiB)
17:38:07.878 INFO MemoryStore - Block broadcast_317_piece0 stored as bytes in memory (estimated size 50.3 KiB, free 1919.3 MiB)
17:38:07.878 INFO BlockManagerInfo - Added broadcast_317_piece0 in memory on localhost:37953 (size: 50.3 KiB, free: 1919.9 MiB)
17:38:07.878 INFO SparkContext - Created broadcast 317 from newAPIHadoopFile at SamSource.java:108
17:38:07.880 INFO MemoryStore - Block broadcast_318 stored as values in memory (estimated size 21.0 KiB, free 1919.2 MiB)
17:38:07.881 INFO MemoryStore - Block broadcast_318_piece0 stored as bytes in memory (estimated size 2.4 KiB, free 1919.2 MiB)
17:38:07.881 INFO BlockManagerInfo - Added broadcast_318_piece0 in memory on localhost:37953 (size: 2.4 KiB, free: 1919.9 MiB)
17:38:07.881 INFO SparkContext - Created broadcast 318 from broadcast at ReadsSparkSink.java:133
17:38:07.882 INFO MemoryStore - Block broadcast_319 stored as values in memory (estimated size 21.5 KiB, free 1919.2 MiB)
17:38:07.882 INFO MemoryStore - Block broadcast_319_piece0 stored as bytes in memory (estimated size 2.4 KiB, free 1919.2 MiB)
17:38:07.882 INFO BlockManagerInfo - Added broadcast_319_piece0 in memory on localhost:37953 (size: 2.4 KiB, free: 1919.9 MiB)
17:38:07.882 INFO SparkContext - Created broadcast 319 from broadcast at BamSink.java:76
17:38:07.884 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:07.884 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:07.884 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:07.897 INFO SparkContext - Starting job: runJob at SparkHadoopWriter.scala:83
17:38:07.898 INFO DAGScheduler - Got job 120 (runJob at SparkHadoopWriter.scala:83) with 1 output partitions
17:38:07.898 INFO DAGScheduler - Final stage: ResultStage 165 (runJob at SparkHadoopWriter.scala:83)
17:38:07.898 INFO DAGScheduler - Parents of final stage: List()
17:38:07.898 INFO DAGScheduler - Missing parents: List()
17:38:07.898 INFO DAGScheduler - Submitting ResultStage 165 (MapPartitionsRDD[770] at mapToPair at BamSink.java:91), which has no missing parents
17:38:07.904 INFO MemoryStore - Block broadcast_320 stored as values in memory (estimated size 152.3 KiB, free 1919.1 MiB)
17:38:07.907 INFO MemoryStore - Block broadcast_320_piece0 stored as bytes in memory (estimated size 56.4 KiB, free 1919.0 MiB)
17:38:07.907 INFO BlockManagerInfo - Added broadcast_320_piece0 in memory on localhost:37953 (size: 56.4 KiB, free: 1919.8 MiB)
17:38:07.907 INFO SparkContext - Created broadcast 320 from broadcast at DAGScheduler.scala:1580
17:38:07.908 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 165 (MapPartitionsRDD[770] at mapToPair at BamSink.java:91) (first 15 tasks are for partitions Vector(0))
17:38:07.908 INFO TaskSchedulerImpl - Adding task set 165.0 with 1 tasks resource profile 0
17:38:07.908 INFO BlockManagerInfo - Removed broadcast_317_piece0 on localhost:37953 in memory (size: 50.3 KiB, free: 1919.9 MiB)
17:38:07.908 INFO BlockManagerInfo - Removed broadcast_315_piece0 on localhost:37953 in memory (size: 3.9 KiB, free: 1919.9 MiB)
17:38:07.908 INFO TaskSetManager - Starting task 0.0 in stage 165.0 (TID 221) (localhost, executor driver, partition 0, PROCESS_LOCAL, 8561 bytes)
17:38:07.909 INFO Executor - Running task 0.0 in stage 165.0 (TID 221)
17:38:07.909 INFO BlockManagerInfo - Removed broadcast_316_piece0 on localhost:37953 in memory (size: 2.4 KiB, free: 1919.9 MiB)
17:38:07.914 INFO NewHadoopRDD - Input split: file:/home/runner/work/gatk/gatk/src/test/resources/org/broadinstitute/hellbender/engine/CEUTrio.HiSeq.WGS.b37.NA12878.20.21.10000000-10000020.with.unmapped.queryname.samtools.sam:0+224884
17:38:07.916 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:07.916 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:07.916 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:07.917 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:07.917 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:07.917 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:07.924 WARN IntelDeflaterFactory - Intel Deflater not supported, using Java.util.zip.Deflater
17:38:07.939 INFO FileOutputCommitter - Saved output of task 'attempt_20251027173807804331631115684321_0770_r_000000_0' to file:/tmp/ReadsSparkSinkNotSorting17877543770359791722.bam.parts/_temporary/0/task_20251027173807804331631115684321_0770_r_000000
17:38:07.939 INFO SparkHadoopMapRedUtil - attempt_20251027173807804331631115684321_0770_r_000000_0: Committed. Elapsed time: 0 ms.
17:38:07.940 INFO Executor - Finished task 0.0 in stage 165.0 (TID 221). 1084 bytes result sent to driver
17:38:07.940 INFO TaskSetManager - Finished task 0.0 in stage 165.0 (TID 221) in 32 ms on localhost (executor driver) (1/1)
17:38:07.940 INFO TaskSchedulerImpl - Removed TaskSet 165.0, whose tasks have all completed, from pool
17:38:07.940 INFO DAGScheduler - ResultStage 165 (runJob at SparkHadoopWriter.scala:83) finished in 0.042 s
17:38:07.940 INFO DAGScheduler - Job 120 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:07.940 INFO TaskSchedulerImpl - Killing all running tasks in stage 165: Stage finished
17:38:07.940 INFO DAGScheduler - Job 120 finished: runJob at SparkHadoopWriter.scala:83, took 0.042923 s
17:38:07.941 INFO SparkHadoopWriter - Start to commit write Job job_20251027173807804331631115684321_0770.
17:38:07.944 INFO SparkHadoopWriter - Write Job job_20251027173807804331631115684321_0770 committed. Elapsed time: 3 ms.
17:38:07.946 WARN IntelDeflaterFactory - Intel Deflater not supported, using Java.util.zip.Deflater
17:38:07.952 INFO HadoopFileSystemWrapper - Concatenating 3 parts to /tmp/ReadsSparkSinkNotSorting17877543770359791722.bam
17:38:07.955 INFO HadoopFileSystemWrapper - Concatenating to /tmp/ReadsSparkSinkNotSorting17877543770359791722.bam done
17:38:07.955 INFO IndexFileMerger - Merging .sbi files in temp directory /tmp/ReadsSparkSinkNotSorting17877543770359791722.bam.parts/ to /tmp/ReadsSparkSinkNotSorting17877543770359791722.bam.sbi
17:38:07.959 INFO IndexFileMerger - Done merging .sbi files
17:38:07.960 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:07.960 INFO MemoryStore - Block broadcast_321 stored as values in memory (estimated size 192.0 B, free 1919.4 MiB)
17:38:07.961 INFO MemoryStore - Block broadcast_321_piece0 stored as bytes in memory (estimated size 127.0 B, free 1919.4 MiB)
17:38:07.961 INFO BlockManagerInfo - Added broadcast_321_piece0 in memory on localhost:37953 (size: 127.0 B, free: 1919.9 MiB)
17:38:07.961 INFO SparkContext - Created broadcast 321 from broadcast at BamSource.java:104
17:38:07.962 INFO MemoryStore - Block broadcast_322 stored as values in memory (estimated size 297.9 KiB, free 1919.1 MiB)
17:38:07.968 INFO MemoryStore - Block broadcast_322_piece0 stored as bytes in memory (estimated size 50.1 KiB, free 1919.0 MiB)
17:38:07.968 INFO BlockManagerInfo - Added broadcast_322_piece0 in memory on localhost:37953 (size: 50.1 KiB, free: 1919.8 MiB)
17:38:07.968 INFO SparkContext - Created broadcast 322 from newAPIHadoopFile at PathSplitSource.java:96
17:38:07.977 INFO FileInputFormat - Total input files to process : 1
17:38:07.990 INFO SparkContext - Starting job: collect at SparkUtils.java:205
17:38:07.990 INFO DAGScheduler - Got job 121 (collect at SparkUtils.java:205) with 1 output partitions
17:38:07.990 INFO DAGScheduler - Final stage: ResultStage 166 (collect at SparkUtils.java:205)
17:38:07.990 INFO DAGScheduler - Parents of final stage: List()
17:38:07.990 INFO DAGScheduler - Missing parents: List()
17:38:07.990 INFO DAGScheduler - Submitting ResultStage 166 (MapPartitionsRDD[777] at mapPartitions at SparkUtils.java:188), which has no missing parents
17:38:07.996 INFO MemoryStore - Block broadcast_323 stored as values in memory (estimated size 148.6 KiB, free 1918.9 MiB)
17:38:07.997 INFO MemoryStore - Block broadcast_323_piece0 stored as bytes in memory (estimated size 54.7 KiB, free 1918.8 MiB)
17:38:07.997 INFO BlockManagerInfo - Added broadcast_323_piece0 in memory on localhost:37953 (size: 54.7 KiB, free: 1919.8 MiB)
17:38:07.997 INFO SparkContext - Created broadcast 323 from broadcast at DAGScheduler.scala:1580
17:38:07.997 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 166 (MapPartitionsRDD[777] at mapPartitions at SparkUtils.java:188) (first 15 tasks are for partitions Vector(0))
17:38:07.997 INFO TaskSchedulerImpl - Adding task set 166.0 with 1 tasks resource profile 0
17:38:07.997 INFO TaskSetManager - Starting task 0.0 in stage 166.0 (TID 222) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7811 bytes)
17:38:07.998 INFO Executor - Running task 0.0 in stage 166.0 (TID 222)
17:38:08.010 INFO NewHadoopRDD - Input split: file:/tmp/ReadsSparkSinkNotSorting17877543770359791722.bam:0+58999
17:38:08.011 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:08.011 INFO Executor - Finished task 0.0 in stage 166.0 (TID 222). 1700 bytes result sent to driver
17:38:08.012 INFO TaskSetManager - Finished task 0.0 in stage 166.0 (TID 222) in 15 ms on localhost (executor driver) (1/1)
17:38:08.012 INFO TaskSchedulerImpl - Removed TaskSet 166.0, whose tasks have all completed, from pool
17:38:08.012 INFO DAGScheduler - ResultStage 166 (collect at SparkUtils.java:205) finished in 0.021 s
17:38:08.012 INFO DAGScheduler - Job 121 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:08.012 INFO TaskSchedulerImpl - Killing all running tasks in stage 166: Stage finished
17:38:08.012 INFO DAGScheduler - Job 121 finished: collect at SparkUtils.java:205, took 0.022252 s
17:38:08.027 INFO SparkContext - Starting job: collect at ReadsSparkSinkUnitTest.java:91
17:38:08.027 INFO DAGScheduler - Got job 122 (collect at ReadsSparkSinkUnitTest.java:91) with 1 output partitions
17:38:08.027 INFO DAGScheduler - Final stage: ResultStage 167 (collect at ReadsSparkSinkUnitTest.java:91)
17:38:08.027 INFO DAGScheduler - Parents of final stage: List()
17:38:08.027 INFO DAGScheduler - Missing parents: List()
17:38:08.027 INFO DAGScheduler - Submitting ResultStage 167 (ZippedPartitionsRDD2[780] at zipPartitions at SparkUtils.java:244), which has no missing parents
17:38:08.033 INFO MemoryStore - Block broadcast_324 stored as values in memory (estimated size 149.8 KiB, free 1918.7 MiB)
17:38:08.033 INFO MemoryStore - Block broadcast_324_piece0 stored as bytes in memory (estimated size 55.2 KiB, free 1918.6 MiB)
17:38:08.033 INFO BlockManagerInfo - Added broadcast_324_piece0 in memory on localhost:37953 (size: 55.2 KiB, free: 1919.7 MiB)
17:38:08.034 INFO SparkContext - Created broadcast 324 from broadcast at DAGScheduler.scala:1580
17:38:08.034 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 167 (ZippedPartitionsRDD2[780] at zipPartitions at SparkUtils.java:244) (first 15 tasks are for partitions Vector(0))
17:38:08.034 INFO TaskSchedulerImpl - Adding task set 167.0 with 1 tasks resource profile 0
17:38:08.034 INFO TaskSetManager - Starting task 0.0 in stage 167.0 (TID 223) (localhost, executor driver, partition 0, PROCESS_LOCAL, 8436 bytes)
17:38:08.034 INFO Executor - Running task 0.0 in stage 167.0 (TID 223)
17:38:08.046 INFO NewHadoopRDD - Input split: file:/tmp/ReadsSparkSinkNotSorting17877543770359791722.bam:0+58999
17:38:08.046 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:08.048 INFO Executor - Finished task 0.0 in stage 167.0 (TID 223). 192451 bytes result sent to driver
17:38:08.048 INFO TaskSetManager - Finished task 0.0 in stage 167.0 (TID 223) in 14 ms on localhost (executor driver) (1/1)
17:38:08.048 INFO TaskSchedulerImpl - Removed TaskSet 167.0, whose tasks have all completed, from pool
17:38:08.049 INFO DAGScheduler - ResultStage 167 (collect at ReadsSparkSinkUnitTest.java:91) finished in 0.022 s
17:38:08.049 INFO DAGScheduler - Job 122 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:08.049 INFO TaskSchedulerImpl - Killing all running tasks in stage 167: Stage finished
17:38:08.049 INFO DAGScheduler - Job 122 finished: collect at ReadsSparkSinkUnitTest.java:91, took 0.021970 s
WARNING 2025-10-27 17:38:08 SamReaderFactory Unable to detect file format from input URL or stream, assuming SAM format.
WARNING 2025-10-27 17:38:08 SamReaderFactory Unable to detect file format from input URL or stream, assuming SAM format.
17:38:08.050 INFO MemoryStore - Block broadcast_325 stored as values in memory (estimated size 21.0 KiB, free 1918.6 MiB)
17:38:08.051 INFO MemoryStore - Block broadcast_325_piece0 stored as bytes in memory (estimated size 2.4 KiB, free 1918.6 MiB)
17:38:08.051 INFO BlockManagerInfo - Added broadcast_325_piece0 in memory on localhost:37953 (size: 2.4 KiB, free: 1919.7 MiB)
17:38:08.051 INFO SparkContext - Created broadcast 325 from broadcast at SamSource.java:78
17:38:08.052 INFO MemoryStore - Block broadcast_326 stored as values in memory (estimated size 298.0 KiB, free 1918.3 MiB)
17:38:08.057 INFO MemoryStore - Block broadcast_326_piece0 stored as bytes in memory (estimated size 50.3 KiB, free 1918.3 MiB)
17:38:08.057 INFO BlockManagerInfo - Added broadcast_326_piece0 in memory on localhost:37953 (size: 50.3 KiB, free: 1919.7 MiB)
17:38:08.058 INFO SparkContext - Created broadcast 326 from newAPIHadoopFile at SamSource.java:108
17:38:08.061 INFO FileInputFormat - Total input files to process : 1
17:38:08.064 INFO SparkContext - Starting job: collect at SparkUtils.java:205
17:38:08.064 INFO DAGScheduler - Got job 123 (collect at SparkUtils.java:205) with 1 output partitions
17:38:08.064 INFO DAGScheduler - Final stage: ResultStage 168 (collect at SparkUtils.java:205)
17:38:08.064 INFO DAGScheduler - Parents of final stage: List()
17:38:08.064 INFO DAGScheduler - Missing parents: List()
17:38:08.064 INFO DAGScheduler - Submitting ResultStage 168 (MapPartitionsRDD[786] at mapPartitions at SparkUtils.java:188), which has no missing parents
17:38:08.065 INFO MemoryStore - Block broadcast_327 stored as values in memory (estimated size 7.9 KiB, free 1918.3 MiB)
17:38:08.068 INFO MemoryStore - Block broadcast_327_piece0 stored as bytes in memory (estimated size 3.9 KiB, free 1918.3 MiB)
17:38:08.068 INFO BlockManagerInfo - Added broadcast_327_piece0 in memory on localhost:37953 (size: 3.9 KiB, free: 1919.7 MiB)
17:38:08.068 INFO SparkContext - Created broadcast 327 from broadcast at DAGScheduler.scala:1580
17:38:08.068 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 168 (MapPartitionsRDD[786] at mapPartitions at SparkUtils.java:188) (first 15 tasks are for partitions Vector(0))
17:38:08.068 INFO TaskSchedulerImpl - Adding task set 168.0 with 1 tasks resource profile 0
17:38:08.069 INFO BlockManagerInfo - Removed broadcast_319_piece0 on localhost:37953 in memory (size: 2.4 KiB, free: 1919.7 MiB)
17:38:08.069 INFO TaskSetManager - Starting task 0.0 in stage 168.0 (TID 224) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7936 bytes)
17:38:08.069 INFO Executor - Running task 0.0 in stage 168.0 (TID 224)
17:38:08.069 INFO BlockManagerInfo - Removed broadcast_324_piece0 on localhost:37953 in memory (size: 55.2 KiB, free: 1919.7 MiB)
17:38:08.070 INFO BlockManagerInfo - Removed broadcast_318_piece0 on localhost:37953 in memory (size: 2.4 KiB, free: 1919.7 MiB)
17:38:08.070 INFO BlockManagerInfo - Removed broadcast_320_piece0 on localhost:37953 in memory (size: 56.4 KiB, free: 1919.8 MiB)
17:38:08.070 INFO NewHadoopRDD - Input split: file:/home/runner/work/gatk/gatk/src/test/resources/org/broadinstitute/hellbender/engine/CEUTrio.HiSeq.WGS.b37.NA12878.20.21.10000000-10000020.with.unmapped.queryname.samtools.sam:0+224884
17:38:08.071 INFO BlockManagerInfo - Removed broadcast_323_piece0 on localhost:37953 in memory (size: 54.7 KiB, free: 1919.8 MiB)
17:38:08.072 INFO Executor - Finished task 0.0 in stage 168.0 (TID 224). 1657 bytes result sent to driver
17:38:08.072 INFO TaskSetManager - Finished task 0.0 in stage 168.0 (TID 224) in 3 ms on localhost (executor driver) (1/1)
17:38:08.072 INFO TaskSchedulerImpl - Removed TaskSet 168.0, whose tasks have all completed, from pool
17:38:08.072 INFO DAGScheduler - ResultStage 168 (collect at SparkUtils.java:205) finished in 0.007 s
17:38:08.073 INFO DAGScheduler - Job 123 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:08.073 INFO TaskSchedulerImpl - Killing all running tasks in stage 168: Stage finished
17:38:08.073 INFO DAGScheduler - Job 123 finished: collect at SparkUtils.java:205, took 0.008586 s
17:38:08.078 INFO SparkContext - Starting job: collect at ReadsSparkSinkUnitTest.java:94
17:38:08.078 INFO DAGScheduler - Got job 124 (collect at ReadsSparkSinkUnitTest.java:94) with 1 output partitions
17:38:08.078 INFO DAGScheduler - Final stage: ResultStage 169 (collect at ReadsSparkSinkUnitTest.java:94)
17:38:08.078 INFO DAGScheduler - Parents of final stage: List()
17:38:08.078 INFO DAGScheduler - Missing parents: List()
17:38:08.078 INFO DAGScheduler - Submitting ResultStage 169 (ZippedPartitionsRDD2[789] at zipPartitions at SparkUtils.java:244), which has no missing parents
17:38:08.079 INFO MemoryStore - Block broadcast_328 stored as values in memory (estimated size 9.6 KiB, free 1918.9 MiB)
17:38:08.079 INFO MemoryStore - Block broadcast_328_piece0 stored as bytes in memory (estimated size 4.4 KiB, free 1918.9 MiB)
17:38:08.079 INFO BlockManagerInfo - Added broadcast_328_piece0 in memory on localhost:37953 (size: 4.4 KiB, free: 1919.8 MiB)
17:38:08.079 INFO SparkContext - Created broadcast 328 from broadcast at DAGScheduler.scala:1580
17:38:08.079 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 169 (ZippedPartitionsRDD2[789] at zipPartitions at SparkUtils.java:244) (first 15 tasks are for partitions Vector(0))
17:38:08.079 INFO TaskSchedulerImpl - Adding task set 169.0 with 1 tasks resource profile 0
17:38:08.080 INFO TaskSetManager - Starting task 0.0 in stage 169.0 (TID 225) (localhost, executor driver, partition 0, PROCESS_LOCAL, 8561 bytes)
17:38:08.080 INFO Executor - Running task 0.0 in stage 169.0 (TID 225)
17:38:08.081 INFO NewHadoopRDD - Input split: file:/home/runner/work/gatk/gatk/src/test/resources/org/broadinstitute/hellbender/engine/CEUTrio.HiSeq.WGS.b37.NA12878.20.21.10000000-10000020.with.unmapped.queryname.samtools.sam:0+224884
17:38:08.086 INFO Executor - Finished task 0.0 in stage 169.0 (TID 225). 192494 bytes result sent to driver
17:38:08.087 INFO TaskSetManager - Finished task 0.0 in stage 169.0 (TID 225) in 7 ms on localhost (executor driver) (1/1)
17:38:08.087 INFO TaskSchedulerImpl - Removed TaskSet 169.0, whose tasks have all completed, from pool
17:38:08.087 INFO DAGScheduler - ResultStage 169 (collect at ReadsSparkSinkUnitTest.java:94) finished in 0.009 s
17:38:08.087 INFO DAGScheduler - Job 124 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:08.087 INFO TaskSchedulerImpl - Killing all running tasks in stage 169: Stage finished
17:38:08.087 INFO DAGScheduler - Job 124 finished: collect at ReadsSparkSinkUnitTest.java:94, took 0.009711 s
17:38:08.093 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:08.095 INFO MemoryStore - Block broadcast_329 stored as values in memory (estimated size 297.9 KiB, free 1918.6 MiB)
17:38:08.100 INFO MemoryStore - Block broadcast_329_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1918.6 MiB)
17:38:08.100 INFO BlockManagerInfo - Added broadcast_329_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.8 MiB)
17:38:08.101 INFO SparkContext - Created broadcast 329 from newAPIHadoopFile at PathSplitSource.java:96
17:38:08.120 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:08.121 INFO MemoryStore - Block broadcast_330 stored as values in memory (estimated size 297.9 KiB, free 1918.3 MiB)
17:38:08.127 INFO MemoryStore - Block broadcast_330_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1918.2 MiB)
17:38:08.127 INFO BlockManagerInfo - Added broadcast_330_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.7 MiB)
17:38:08.127 INFO SparkContext - Created broadcast 330 from newAPIHadoopFile at PathSplitSource.java:96
17:38:08.146 INFO FileInputFormat - Total input files to process : 1
17:38:08.148 INFO MemoryStore - Block broadcast_331 stored as values in memory (estimated size 160.7 KiB, free 1918.1 MiB)
17:38:08.149 INFO MemoryStore - Block broadcast_331_piece0 stored as bytes in memory (estimated size 9.6 KiB, free 1918.1 MiB)
17:38:08.149 INFO BlockManagerInfo - Added broadcast_331_piece0 in memory on localhost:37953 (size: 9.6 KiB, free: 1919.7 MiB)
17:38:08.150 INFO SparkContext - Created broadcast 331 from broadcast at ReadsSparkSink.java:133
17:38:08.151 INFO MemoryStore - Block broadcast_332 stored as values in memory (estimated size 163.2 KiB, free 1917.9 MiB)
17:38:08.151 INFO MemoryStore - Block broadcast_332_piece0 stored as bytes in memory (estimated size 9.6 KiB, free 1917.9 MiB)
17:38:08.152 INFO BlockManagerInfo - Added broadcast_332_piece0 in memory on localhost:37953 (size: 9.6 KiB, free: 1919.7 MiB)
17:38:08.152 INFO SparkContext - Created broadcast 332 from broadcast at BamSink.java:76
17:38:08.154 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:08.154 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:08.154 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:08.167 INFO SparkContext - Starting job: runJob at SparkHadoopWriter.scala:83
17:38:08.167 INFO DAGScheduler - Registering RDD 803 (mapToPair at SparkUtils.java:161) as input to shuffle 34
17:38:08.167 INFO DAGScheduler - Got job 125 (runJob at SparkHadoopWriter.scala:83) with 1 output partitions
17:38:08.167 INFO DAGScheduler - Final stage: ResultStage 171 (runJob at SparkHadoopWriter.scala:83)
17:38:08.167 INFO DAGScheduler - Parents of final stage: List(ShuffleMapStage 170)
17:38:08.167 INFO DAGScheduler - Missing parents: List(ShuffleMapStage 170)
17:38:08.167 INFO DAGScheduler - Submitting ShuffleMapStage 170 (MapPartitionsRDD[803] at mapToPair at SparkUtils.java:161), which has no missing parents
17:38:08.183 INFO MemoryStore - Block broadcast_333 stored as values in memory (estimated size 520.4 KiB, free 1917.4 MiB)
17:38:08.184 INFO MemoryStore - Block broadcast_333_piece0 stored as bytes in memory (estimated size 166.1 KiB, free 1917.2 MiB)
17:38:08.184 INFO BlockManagerInfo - Added broadcast_333_piece0 in memory on localhost:37953 (size: 166.1 KiB, free: 1919.6 MiB)
17:38:08.185 INFO SparkContext - Created broadcast 333 from broadcast at DAGScheduler.scala:1580
17:38:08.185 INFO DAGScheduler - Submitting 1 missing tasks from ShuffleMapStage 170 (MapPartitionsRDD[803] at mapToPair at SparkUtils.java:161) (first 15 tasks are for partitions Vector(0))
17:38:08.185 INFO TaskSchedulerImpl - Adding task set 170.0 with 1 tasks resource profile 0
17:38:08.185 INFO TaskSetManager - Starting task 0.0 in stage 170.0 (TID 226) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7861 bytes)
17:38:08.186 INFO Executor - Running task 0.0 in stage 170.0 (TID 226)
17:38:08.218 INFO NewHadoopRDD - Input split: file:/home/runner/work/gatk/gatk/src/test/resources/org/broadinstitute/hellbender/tools/BQSR/HiSeq.1mb.1RG.2k_lines.bam:0+222075
17:38:08.219 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:08.220 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:08.227 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:08.238 INFO Executor - Finished task 0.0 in stage 170.0 (TID 226). 1234 bytes result sent to driver
17:38:08.239 INFO BlockManagerInfo - Removed broadcast_313_piece0 on localhost:37953 in memory (size: 2.4 KiB, free: 1919.6 MiB)
17:38:08.239 INFO TaskSetManager - Finished task 0.0 in stage 170.0 (TID 226) in 54 ms on localhost (executor driver) (1/1)
17:38:08.239 INFO TaskSchedulerImpl - Removed TaskSet 170.0, whose tasks have all completed, from pool
17:38:08.239 INFO DAGScheduler - ShuffleMapStage 170 (mapToPair at SparkUtils.java:161) finished in 0.071 s
17:38:08.239 INFO DAGScheduler - looking for newly runnable stages
17:38:08.239 INFO DAGScheduler - running: HashSet()
17:38:08.239 INFO DAGScheduler - waiting: HashSet(ResultStage 171)
17:38:08.239 INFO DAGScheduler - failed: HashSet()
17:38:08.239 INFO DAGScheduler - Submitting ResultStage 171 (MapPartitionsRDD[808] at mapToPair at BamSink.java:91), which has no missing parents
17:38:08.239 INFO BlockManagerInfo - Removed broadcast_327_piece0 on localhost:37953 in memory (size: 3.9 KiB, free: 1919.6 MiB)
17:38:08.240 INFO BlockManagerInfo - Removed broadcast_322_piece0 on localhost:37953 in memory (size: 50.1 KiB, free: 1919.6 MiB)
17:38:08.240 INFO BlockManagerInfo - Removed broadcast_328_piece0 on localhost:37953 in memory (size: 4.4 KiB, free: 1919.6 MiB)
17:38:08.241 INFO BlockManagerInfo - Removed broadcast_330_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.7 MiB)
17:38:08.241 INFO BlockManagerInfo - Removed broadcast_325_piece0 on localhost:37953 in memory (size: 2.4 KiB, free: 1919.7 MiB)
17:38:08.242 INFO BlockManagerInfo - Removed broadcast_314_piece0 on localhost:37953 in memory (size: 50.3 KiB, free: 1919.7 MiB)
17:38:08.242 INFO BlockManagerInfo - Removed broadcast_321_piece0 on localhost:37953 in memory (size: 127.0 B, free: 1919.7 MiB)
17:38:08.242 INFO BlockManagerInfo - Removed broadcast_326_piece0 on localhost:37953 in memory (size: 50.3 KiB, free: 1919.8 MiB)
17:38:08.245 INFO MemoryStore - Block broadcast_334 stored as values in memory (estimated size 241.4 KiB, free 1918.4 MiB)
17:38:08.246 INFO MemoryStore - Block broadcast_334_piece0 stored as bytes in memory (estimated size 67.0 KiB, free 1918.4 MiB)
17:38:08.246 INFO BlockManagerInfo - Added broadcast_334_piece0 in memory on localhost:37953 (size: 67.0 KiB, free: 1919.7 MiB)
17:38:08.246 INFO SparkContext - Created broadcast 334 from broadcast at DAGScheduler.scala:1580
17:38:08.247 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 171 (MapPartitionsRDD[808] at mapToPair at BamSink.java:91) (first 15 tasks are for partitions Vector(0))
17:38:08.247 INFO TaskSchedulerImpl - Adding task set 171.0 with 1 tasks resource profile 0
17:38:08.247 INFO TaskSetManager - Starting task 0.0 in stage 171.0 (TID 227) (localhost, executor driver, partition 0, NODE_LOCAL, 7513 bytes)
17:38:08.247 INFO Executor - Running task 0.0 in stage 171.0 (TID 227)
17:38:08.252 INFO ShuffleBlockFetcherIterator - Getting 1 (343.8 KiB) non-empty blocks including 1 (343.8 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
17:38:08.252 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
17:38:08.264 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:08.264 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:08.264 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:08.264 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:08.264 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:08.264 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:08.271 WARN IntelDeflaterFactory - Intel Deflater not supported, using Java.util.zip.Deflater
17:38:08.288 INFO FileOutputCommitter - Saved output of task 'attempt_202510271738081373126064768822673_0808_r_000000_0' to file:/tmp/ReadsSparkSinkUnitTest1.someOtherPlace8847643063146886025/_temporary/0/task_202510271738081373126064768822673_0808_r_000000
17:38:08.288 INFO SparkHadoopMapRedUtil - attempt_202510271738081373126064768822673_0808_r_000000_0: Committed. Elapsed time: 0 ms.
17:38:08.288 INFO Executor - Finished task 0.0 in stage 171.0 (TID 227). 1858 bytes result sent to driver
17:38:08.288 INFO TaskSetManager - Finished task 0.0 in stage 171.0 (TID 227) in 41 ms on localhost (executor driver) (1/1)
17:38:08.288 INFO TaskSchedulerImpl - Removed TaskSet 171.0, whose tasks have all completed, from pool
17:38:08.289 INFO DAGScheduler - ResultStage 171 (runJob at SparkHadoopWriter.scala:83) finished in 0.050 s
17:38:08.289 INFO DAGScheduler - Job 125 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:08.289 INFO TaskSchedulerImpl - Killing all running tasks in stage 171: Stage finished
17:38:08.289 INFO DAGScheduler - Job 125 finished: runJob at SparkHadoopWriter.scala:83, took 0.122212 s
17:38:08.289 INFO SparkHadoopWriter - Start to commit write Job job_202510271738081373126064768822673_0808.
17:38:08.293 INFO SparkHadoopWriter - Write Job job_202510271738081373126064768822673_0808 committed. Elapsed time: 3 ms.
17:38:08.295 WARN IntelDeflaterFactory - Intel Deflater not supported, using Java.util.zip.Deflater
17:38:08.301 INFO HadoopFileSystemWrapper - Concatenating 3 parts to /tmp/ReadsSparkSinkUnitTest114735412111205098522.bam
17:38:08.304 INFO HadoopFileSystemWrapper - Concatenating to /tmp/ReadsSparkSinkUnitTest114735412111205098522.bam done
17:38:08.304 INFO IndexFileMerger - Merging .sbi files in temp directory /tmp/ReadsSparkSinkUnitTest1.someOtherPlace8847643063146886025 to /tmp/ReadsSparkSinkUnitTest114735412111205098522.bam.sbi
17:38:08.308 INFO IndexFileMerger - Done merging .sbi files
17:38:08.308 INFO IndexFileMerger - Merging .bai files in temp directory /tmp/ReadsSparkSinkUnitTest1.someOtherPlace8847643063146886025 to /tmp/ReadsSparkSinkUnitTest114735412111205098522.bam.bai
17:38:08.312 INFO IndexFileMerger - Done merging .bai files
17:38:08.313 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:08.314 INFO MemoryStore - Block broadcast_335 stored as values in memory (estimated size 320.0 B, free 1918.4 MiB)
17:38:08.314 INFO MemoryStore - Block broadcast_335_piece0 stored as bytes in memory (estimated size 233.0 B, free 1918.4 MiB)
17:38:08.314 INFO BlockManagerInfo - Added broadcast_335_piece0 in memory on localhost:37953 (size: 233.0 B, free: 1919.7 MiB)
17:38:08.315 INFO SparkContext - Created broadcast 335 from broadcast at BamSource.java:104
17:38:08.316 INFO MemoryStore - Block broadcast_336 stored as values in memory (estimated size 297.9 KiB, free 1918.1 MiB)
17:38:08.321 INFO MemoryStore - Block broadcast_336_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1918.0 MiB)
17:38:08.321 INFO BlockManagerInfo - Added broadcast_336_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.7 MiB)
17:38:08.321 INFO SparkContext - Created broadcast 336 from newAPIHadoopFile at PathSplitSource.java:96
17:38:08.330 INFO FileInputFormat - Total input files to process : 1
17:38:08.343 INFO SparkContext - Starting job: collect at ReadsSparkSinkUnitTest.java:182
17:38:08.343 INFO DAGScheduler - Got job 126 (collect at ReadsSparkSinkUnitTest.java:182) with 1 output partitions
17:38:08.343 INFO DAGScheduler - Final stage: ResultStage 172 (collect at ReadsSparkSinkUnitTest.java:182)
17:38:08.343 INFO DAGScheduler - Parents of final stage: List()
17:38:08.343 INFO DAGScheduler - Missing parents: List()
17:38:08.343 INFO DAGScheduler - Submitting ResultStage 172 (MapPartitionsRDD[814] at filter at ReadsSparkSource.java:96), which has no missing parents
17:38:08.349 INFO MemoryStore - Block broadcast_337 stored as values in memory (estimated size 148.2 KiB, free 1917.9 MiB)
17:38:08.349 INFO MemoryStore - Block broadcast_337_piece0 stored as bytes in memory (estimated size 54.5 KiB, free 1917.8 MiB)
17:38:08.350 INFO BlockManagerInfo - Added broadcast_337_piece0 in memory on localhost:37953 (size: 54.5 KiB, free: 1919.6 MiB)
17:38:08.350 INFO SparkContext - Created broadcast 337 from broadcast at DAGScheduler.scala:1580
17:38:08.350 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 172 (MapPartitionsRDD[814] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:38:08.350 INFO TaskSchedulerImpl - Adding task set 172.0 with 1 tasks resource profile 0
17:38:08.351 INFO TaskSetManager - Starting task 0.0 in stage 172.0 (TID 228) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7810 bytes)
17:38:08.351 INFO Executor - Running task 0.0 in stage 172.0 (TID 228)
17:38:08.363 INFO NewHadoopRDD - Input split: file:/tmp/ReadsSparkSinkUnitTest114735412111205098522.bam:0+238969
17:38:08.364 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:08.368 INFO Executor - Finished task 0.0 in stage 172.0 (TID 228). 651483 bytes result sent to driver
17:38:08.370 INFO TaskSetManager - Finished task 0.0 in stage 172.0 (TID 228) in 20 ms on localhost (executor driver) (1/1)
17:38:08.370 INFO TaskSchedulerImpl - Removed TaskSet 172.0, whose tasks have all completed, from pool
17:38:08.370 INFO DAGScheduler - ResultStage 172 (collect at ReadsSparkSinkUnitTest.java:182) finished in 0.026 s
17:38:08.370 INFO DAGScheduler - Job 126 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:08.370 INFO TaskSchedulerImpl - Killing all running tasks in stage 172: Stage finished
17:38:08.370 INFO DAGScheduler - Job 126 finished: collect at ReadsSparkSinkUnitTest.java:182, took 0.027065 s
17:38:08.380 INFO SparkContext - Starting job: count at ReadsSparkSinkUnitTest.java:185
17:38:08.380 INFO DAGScheduler - Got job 127 (count at ReadsSparkSinkUnitTest.java:185) with 1 output partitions
17:38:08.380 INFO DAGScheduler - Final stage: ResultStage 173 (count at ReadsSparkSinkUnitTest.java:185)
17:38:08.380 INFO DAGScheduler - Parents of final stage: List()
17:38:08.380 INFO DAGScheduler - Missing parents: List()
17:38:08.380 INFO DAGScheduler - Submitting ResultStage 173 (MapPartitionsRDD[796] at filter at ReadsSparkSource.java:96), which has no missing parents
17:38:08.395 INFO MemoryStore - Block broadcast_338 stored as values in memory (estimated size 426.1 KiB, free 1917.4 MiB)
17:38:08.396 INFO MemoryStore - Block broadcast_338_piece0 stored as bytes in memory (estimated size 153.6 KiB, free 1917.2 MiB)
17:38:08.397 INFO BlockManagerInfo - Added broadcast_338_piece0 in memory on localhost:37953 (size: 153.6 KiB, free: 1919.5 MiB)
17:38:08.397 INFO SparkContext - Created broadcast 338 from broadcast at DAGScheduler.scala:1580
17:38:08.397 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 173 (MapPartitionsRDD[796] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:38:08.397 INFO TaskSchedulerImpl - Adding task set 173.0 with 1 tasks resource profile 0
17:38:08.397 INFO TaskSetManager - Starting task 0.0 in stage 173.0 (TID 229) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7872 bytes)
17:38:08.398 INFO Executor - Running task 0.0 in stage 173.0 (TID 229)
17:38:08.428 INFO NewHadoopRDD - Input split: file:/home/runner/work/gatk/gatk/src/test/resources/org/broadinstitute/hellbender/tools/BQSR/HiSeq.1mb.1RG.2k_lines.bam:0+222075
17:38:08.429 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:08.430 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:08.437 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:08.440 INFO Executor - Finished task 0.0 in stage 173.0 (TID 229). 989 bytes result sent to driver
17:38:08.440 INFO TaskSetManager - Finished task 0.0 in stage 173.0 (TID 229) in 43 ms on localhost (executor driver) (1/1)
17:38:08.441 INFO TaskSchedulerImpl - Removed TaskSet 173.0, whose tasks have all completed, from pool
17:38:08.441 INFO DAGScheduler - ResultStage 173 (count at ReadsSparkSinkUnitTest.java:185) finished in 0.061 s
17:38:08.441 INFO DAGScheduler - Job 127 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:08.441 INFO TaskSchedulerImpl - Killing all running tasks in stage 173: Stage finished
17:38:08.441 INFO DAGScheduler - Job 127 finished: count at ReadsSparkSinkUnitTest.java:185, took 0.061220 s
17:38:08.444 INFO SparkContext - Starting job: count at ReadsSparkSinkUnitTest.java:185
17:38:08.444 INFO DAGScheduler - Got job 128 (count at ReadsSparkSinkUnitTest.java:185) with 1 output partitions
17:38:08.444 INFO DAGScheduler - Final stage: ResultStage 174 (count at ReadsSparkSinkUnitTest.java:185)
17:38:08.444 INFO DAGScheduler - Parents of final stage: List()
17:38:08.444 INFO DAGScheduler - Missing parents: List()
17:38:08.444 INFO DAGScheduler - Submitting ResultStage 174 (MapPartitionsRDD[814] at filter at ReadsSparkSource.java:96), which has no missing parents
17:38:08.450 INFO MemoryStore - Block broadcast_339 stored as values in memory (estimated size 148.1 KiB, free 1917.1 MiB)
17:38:08.451 INFO MemoryStore - Block broadcast_339_piece0 stored as bytes in memory (estimated size 54.5 KiB, free 1917.1 MiB)
17:38:08.451 INFO BlockManagerInfo - Added broadcast_339_piece0 in memory on localhost:37953 (size: 54.5 KiB, free: 1919.4 MiB)
17:38:08.451 INFO SparkContext - Created broadcast 339 from broadcast at DAGScheduler.scala:1580
17:38:08.451 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 174 (MapPartitionsRDD[814] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:38:08.451 INFO TaskSchedulerImpl - Adding task set 174.0 with 1 tasks resource profile 0
17:38:08.451 INFO TaskSetManager - Starting task 0.0 in stage 174.0 (TID 230) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7810 bytes)
17:38:08.452 INFO Executor - Running task 0.0 in stage 174.0 (TID 230)
17:38:08.462 INFO NewHadoopRDD - Input split: file:/tmp/ReadsSparkSinkUnitTest114735412111205098522.bam:0+238969
17:38:08.463 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:08.469 INFO Executor - Finished task 0.0 in stage 174.0 (TID 230). 1075 bytes result sent to driver
17:38:08.470 INFO BlockManagerInfo - Removed broadcast_338_piece0 on localhost:37953 in memory (size: 153.6 KiB, free: 1919.5 MiB)
17:38:08.470 INFO TaskSetManager - Finished task 0.0 in stage 174.0 (TID 230) in 19 ms on localhost (executor driver) (1/1)
17:38:08.470 INFO TaskSchedulerImpl - Removed TaskSet 174.0, whose tasks have all completed, from pool
17:38:08.470 INFO DAGScheduler - ResultStage 174 (count at ReadsSparkSinkUnitTest.java:185) finished in 0.025 s
17:38:08.470 INFO DAGScheduler - Job 128 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:08.470 INFO TaskSchedulerImpl - Killing all running tasks in stage 174: Stage finished
17:38:08.470 INFO BlockManagerInfo - Removed broadcast_334_piece0 on localhost:37953 in memory (size: 67.0 KiB, free: 1919.6 MiB)
17:38:08.470 INFO DAGScheduler - Job 128 finished: count at ReadsSparkSinkUnitTest.java:185, took 0.026224 s
17:38:08.471 INFO BlockManagerInfo - Removed broadcast_333_piece0 on localhost:37953 in memory (size: 166.1 KiB, free: 1919.8 MiB)
17:38:08.471 INFO BlockManagerInfo - Removed broadcast_337_piece0 on localhost:37953 in memory (size: 54.5 KiB, free: 1919.8 MiB)
17:38:08.472 INFO BlockManagerInfo - Removed broadcast_331_piece0 on localhost:37953 in memory (size: 9.6 KiB, free: 1919.8 MiB)
17:38:08.473 INFO BlockManagerInfo - Removed broadcast_332_piece0 on localhost:37953 in memory (size: 9.6 KiB, free: 1919.8 MiB)
17:38:08.475 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:08.477 INFO MemoryStore - Block broadcast_340 stored as values in memory (estimated size 297.9 KiB, free 1918.8 MiB)
17:38:08.482 INFO MemoryStore - Block broadcast_340_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1918.8 MiB)
17:38:08.483 INFO BlockManagerInfo - Added broadcast_340_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.8 MiB)
17:38:08.483 INFO SparkContext - Created broadcast 340 from newAPIHadoopFile at PathSplitSource.java:96
17:38:08.502 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:08.504 INFO MemoryStore - Block broadcast_341 stored as values in memory (estimated size 297.9 KiB, free 1918.5 MiB)
17:38:08.510 INFO MemoryStore - Block broadcast_341_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1918.4 MiB)
17:38:08.510 INFO BlockManagerInfo - Added broadcast_341_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.8 MiB)
17:38:08.510 INFO SparkContext - Created broadcast 341 from newAPIHadoopFile at PathSplitSource.java:96
17:38:08.529 INFO FileInputFormat - Total input files to process : 1
17:38:08.531 INFO MemoryStore - Block broadcast_342 stored as values in memory (estimated size 160.7 KiB, free 1918.3 MiB)
17:38:08.531 INFO MemoryStore - Block broadcast_342_piece0 stored as bytes in memory (estimated size 9.6 KiB, free 1918.3 MiB)
17:38:08.532 INFO BlockManagerInfo - Added broadcast_342_piece0 in memory on localhost:37953 (size: 9.6 KiB, free: 1919.7 MiB)
17:38:08.532 INFO SparkContext - Created broadcast 342 from broadcast at ReadsSparkSink.java:133
17:38:08.533 INFO MemoryStore - Block broadcast_343 stored as values in memory (estimated size 163.2 KiB, free 1918.1 MiB)
17:38:08.534 INFO MemoryStore - Block broadcast_343_piece0 stored as bytes in memory (estimated size 9.6 KiB, free 1918.1 MiB)
17:38:08.534 INFO BlockManagerInfo - Added broadcast_343_piece0 in memory on localhost:37953 (size: 9.6 KiB, free: 1919.7 MiB)
17:38:08.534 INFO SparkContext - Created broadcast 343 from broadcast at BamSink.java:76
17:38:08.536 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:08.536 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:08.536 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:08.549 INFO SparkContext - Starting job: runJob at SparkHadoopWriter.scala:83
17:38:08.549 INFO DAGScheduler - Registering RDD 828 (mapToPair at SparkUtils.java:161) as input to shuffle 35
17:38:08.549 INFO DAGScheduler - Got job 129 (runJob at SparkHadoopWriter.scala:83) with 1 output partitions
17:38:08.549 INFO DAGScheduler - Final stage: ResultStage 176 (runJob at SparkHadoopWriter.scala:83)
17:38:08.549 INFO DAGScheduler - Parents of final stage: List(ShuffleMapStage 175)
17:38:08.549 INFO DAGScheduler - Missing parents: List(ShuffleMapStage 175)
17:38:08.550 INFO DAGScheduler - Submitting ShuffleMapStage 175 (MapPartitionsRDD[828] at mapToPair at SparkUtils.java:161), which has no missing parents
17:38:08.565 INFO MemoryStore - Block broadcast_344 stored as values in memory (estimated size 520.4 KiB, free 1917.6 MiB)
17:38:08.566 INFO MemoryStore - Block broadcast_344_piece0 stored as bytes in memory (estimated size 166.1 KiB, free 1917.4 MiB)
17:38:08.566 INFO BlockManagerInfo - Added broadcast_344_piece0 in memory on localhost:37953 (size: 166.1 KiB, free: 1919.6 MiB)
17:38:08.567 INFO SparkContext - Created broadcast 344 from broadcast at DAGScheduler.scala:1580
17:38:08.567 INFO DAGScheduler - Submitting 1 missing tasks from ShuffleMapStage 175 (MapPartitionsRDD[828] at mapToPair at SparkUtils.java:161) (first 15 tasks are for partitions Vector(0))
17:38:08.567 INFO TaskSchedulerImpl - Adding task set 175.0 with 1 tasks resource profile 0
17:38:08.567 INFO TaskSetManager - Starting task 0.0 in stage 175.0 (TID 231) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7861 bytes)
17:38:08.568 INFO Executor - Running task 0.0 in stage 175.0 (TID 231)
17:38:08.600 INFO NewHadoopRDD - Input split: file:/home/runner/work/gatk/gatk/src/test/resources/org/broadinstitute/hellbender/tools/BQSR/HiSeq.1mb.1RG.2k_lines.bam:0+222075
17:38:08.601 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:08.602 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:08.609 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:08.617 INFO Executor - Finished task 0.0 in stage 175.0 (TID 231). 1148 bytes result sent to driver
17:38:08.618 INFO TaskSetManager - Finished task 0.0 in stage 175.0 (TID 231) in 51 ms on localhost (executor driver) (1/1)
17:38:08.618 INFO TaskSchedulerImpl - Removed TaskSet 175.0, whose tasks have all completed, from pool
17:38:08.618 INFO DAGScheduler - ShuffleMapStage 175 (mapToPair at SparkUtils.java:161) finished in 0.068 s
17:38:08.618 INFO DAGScheduler - looking for newly runnable stages
17:38:08.618 INFO DAGScheduler - running: HashSet()
17:38:08.618 INFO DAGScheduler - waiting: HashSet(ResultStage 176)
17:38:08.618 INFO DAGScheduler - failed: HashSet()
17:38:08.618 INFO DAGScheduler - Submitting ResultStage 176 (MapPartitionsRDD[833] at mapToPair at BamSink.java:91), which has no missing parents
17:38:08.624 INFO MemoryStore - Block broadcast_345 stored as values in memory (estimated size 241.4 KiB, free 1917.2 MiB)
17:38:08.625 INFO MemoryStore - Block broadcast_345_piece0 stored as bytes in memory (estimated size 67.0 KiB, free 1917.1 MiB)
17:38:08.625 INFO BlockManagerInfo - Added broadcast_345_piece0 in memory on localhost:37953 (size: 67.0 KiB, free: 1919.5 MiB)
17:38:08.625 INFO SparkContext - Created broadcast 345 from broadcast at DAGScheduler.scala:1580
17:38:08.625 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 176 (MapPartitionsRDD[833] at mapToPair at BamSink.java:91) (first 15 tasks are for partitions Vector(0))
17:38:08.625 INFO TaskSchedulerImpl - Adding task set 176.0 with 1 tasks resource profile 0
17:38:08.626 INFO TaskSetManager - Starting task 0.0 in stage 176.0 (TID 232) (localhost, executor driver, partition 0, NODE_LOCAL, 7513 bytes)
17:38:08.626 INFO Executor - Running task 0.0 in stage 176.0 (TID 232)
17:38:08.631 INFO ShuffleBlockFetcherIterator - Getting 1 (343.8 KiB) non-empty blocks including 1 (343.8 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
17:38:08.631 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
17:38:08.642 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:08.642 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:08.642 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:08.642 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:08.643 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:08.643 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:08.650 WARN IntelDeflaterFactory - Intel Deflater not supported, using Java.util.zip.Deflater
17:38:08.665 INFO FileOutputCommitter - Saved output of task 'attempt_202510271738081287635269248595540_0833_r_000000_0' to file:/tmp/ReadsSparkSinkUnitTest1.someOtherPlace1540099052906535353/_temporary/0/task_202510271738081287635269248595540_0833_r_000000
17:38:08.665 INFO SparkHadoopMapRedUtil - attempt_202510271738081287635269248595540_0833_r_000000_0: Committed. Elapsed time: 0 ms.
17:38:08.666 INFO Executor - Finished task 0.0 in stage 176.0 (TID 232). 1858 bytes result sent to driver
17:38:08.666 INFO TaskSetManager - Finished task 0.0 in stage 176.0 (TID 232) in 40 ms on localhost (executor driver) (1/1)
17:38:08.666 INFO TaskSchedulerImpl - Removed TaskSet 176.0, whose tasks have all completed, from pool
17:38:08.666 INFO DAGScheduler - ResultStage 176 (runJob at SparkHadoopWriter.scala:83) finished in 0.047 s
17:38:08.666 INFO DAGScheduler - Job 129 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:08.666 INFO TaskSchedulerImpl - Killing all running tasks in stage 176: Stage finished
17:38:08.666 INFO DAGScheduler - Job 129 finished: runJob at SparkHadoopWriter.scala:83, took 0.117721 s
17:38:08.667 INFO SparkHadoopWriter - Start to commit write Job job_202510271738081287635269248595540_0833.
17:38:08.670 INFO SparkHadoopWriter - Write Job job_202510271738081287635269248595540_0833 committed. Elapsed time: 3 ms.
17:38:08.672 WARN IntelDeflaterFactory - Intel Deflater not supported, using Java.util.zip.Deflater
17:38:08.678 INFO HadoopFileSystemWrapper - Concatenating 3 parts to /tmp/ReadsSparkSinkUnitTest112110568822137566832.bam
17:38:08.681 INFO HadoopFileSystemWrapper - Concatenating to /tmp/ReadsSparkSinkUnitTest112110568822137566832.bam done
17:38:08.681 INFO IndexFileMerger - Merging .sbi files in temp directory /tmp/ReadsSparkSinkUnitTest1.someOtherPlace1540099052906535353 to /tmp/ReadsSparkSinkUnitTest112110568822137566832.bam.sbi
17:38:08.685 INFO IndexFileMerger - Done merging .sbi files
17:38:08.685 INFO IndexFileMerger - Merging .bai files in temp directory /tmp/ReadsSparkSinkUnitTest1.someOtherPlace1540099052906535353 to /tmp/ReadsSparkSinkUnitTest112110568822137566832.bam.bai
17:38:08.689 INFO IndexFileMerger - Done merging .bai files
17:38:08.690 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:08.691 INFO MemoryStore - Block broadcast_346 stored as values in memory (estimated size 13.3 KiB, free 1917.1 MiB)
17:38:08.695 INFO MemoryStore - Block broadcast_346_piece0 stored as bytes in memory (estimated size 8.3 KiB, free 1917.1 MiB)
17:38:08.695 INFO BlockManagerInfo - Added broadcast_346_piece0 in memory on localhost:37953 (size: 8.3 KiB, free: 1919.5 MiB)
17:38:08.695 INFO BlockManagerInfo - Removed broadcast_345_piece0 on localhost:37953 in memory (size: 67.0 KiB, free: 1919.6 MiB)
17:38:08.695 INFO SparkContext - Created broadcast 346 from broadcast at BamSource.java:104
17:38:08.696 INFO MemoryStore - Block broadcast_347 stored as values in memory (estimated size 297.9 KiB, free 1916.9 MiB)
17:38:08.698 INFO BlockManagerInfo - Removed broadcast_336_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.6 MiB)
17:38:08.698 INFO BlockManagerInfo - Removed broadcast_339_piece0 on localhost:37953 in memory (size: 54.5 KiB, free: 1919.7 MiB)
17:38:08.699 INFO BlockManagerInfo - Removed broadcast_343_piece0 on localhost:37953 in memory (size: 9.6 KiB, free: 1919.7 MiB)
17:38:08.699 INFO BlockManagerInfo - Removed broadcast_342_piece0 on localhost:37953 in memory (size: 9.6 KiB, free: 1919.7 MiB)
17:38:08.700 INFO BlockManagerInfo - Removed broadcast_329_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.7 MiB)
17:38:08.701 INFO BlockManagerInfo - Removed broadcast_341_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.8 MiB)
17:38:08.701 INFO BlockManagerInfo - Removed broadcast_335_piece0 on localhost:37953 in memory (size: 233.0 B, free: 1919.8 MiB)
17:38:08.702 INFO MemoryStore - Block broadcast_347_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1918.6 MiB)
17:38:08.702 INFO BlockManagerInfo - Added broadcast_347_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.7 MiB)
17:38:08.702 INFO SparkContext - Created broadcast 347 from newAPIHadoopFile at PathSplitSource.java:96
17:38:08.702 INFO BlockManagerInfo - Removed broadcast_344_piece0 on localhost:37953 in memory (size: 166.1 KiB, free: 1919.9 MiB)
17:38:08.711 INFO FileInputFormat - Total input files to process : 1
17:38:08.724 INFO SparkContext - Starting job: collect at ReadsSparkSinkUnitTest.java:182
17:38:08.724 INFO DAGScheduler - Got job 130 (collect at ReadsSparkSinkUnitTest.java:182) with 1 output partitions
17:38:08.724 INFO DAGScheduler - Final stage: ResultStage 177 (collect at ReadsSparkSinkUnitTest.java:182)
17:38:08.724 INFO DAGScheduler - Parents of final stage: List()
17:38:08.724 INFO DAGScheduler - Missing parents: List()
17:38:08.724 INFO DAGScheduler - Submitting ResultStage 177 (MapPartitionsRDD[839] at filter at ReadsSparkSource.java:96), which has no missing parents
17:38:08.730 INFO MemoryStore - Block broadcast_348 stored as values in memory (estimated size 148.2 KiB, free 1919.2 MiB)
17:38:08.730 INFO MemoryStore - Block broadcast_348_piece0 stored as bytes in memory (estimated size 54.6 KiB, free 1919.1 MiB)
17:38:08.731 INFO BlockManagerInfo - Added broadcast_348_piece0 in memory on localhost:37953 (size: 54.6 KiB, free: 1919.8 MiB)
17:38:08.731 INFO SparkContext - Created broadcast 348 from broadcast at DAGScheduler.scala:1580
17:38:08.731 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 177 (MapPartitionsRDD[839] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:38:08.731 INFO TaskSchedulerImpl - Adding task set 177.0 with 1 tasks resource profile 0
17:38:08.732 INFO TaskSetManager - Starting task 0.0 in stage 177.0 (TID 233) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7810 bytes)
17:38:08.732 INFO Executor - Running task 0.0 in stage 177.0 (TID 233)
17:38:08.744 INFO NewHadoopRDD - Input split: file:/tmp/ReadsSparkSinkUnitTest112110568822137566832.bam:0+238969
17:38:08.745 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:08.749 INFO Executor - Finished task 0.0 in stage 177.0 (TID 233). 651483 bytes result sent to driver
17:38:08.751 INFO TaskSetManager - Finished task 0.0 in stage 177.0 (TID 233) in 20 ms on localhost (executor driver) (1/1)
17:38:08.751 INFO TaskSchedulerImpl - Removed TaskSet 177.0, whose tasks have all completed, from pool
17:38:08.751 INFO DAGScheduler - ResultStage 177 (collect at ReadsSparkSinkUnitTest.java:182) finished in 0.026 s
17:38:08.751 INFO DAGScheduler - Job 130 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:08.751 INFO TaskSchedulerImpl - Killing all running tasks in stage 177: Stage finished
17:38:08.751 INFO DAGScheduler - Job 130 finished: collect at ReadsSparkSinkUnitTest.java:182, took 0.027277 s
17:38:08.761 INFO SparkContext - Starting job: count at ReadsSparkSinkUnitTest.java:185
17:38:08.761 INFO DAGScheduler - Got job 131 (count at ReadsSparkSinkUnitTest.java:185) with 1 output partitions
17:38:08.761 INFO DAGScheduler - Final stage: ResultStage 178 (count at ReadsSparkSinkUnitTest.java:185)
17:38:08.761 INFO DAGScheduler - Parents of final stage: List()
17:38:08.761 INFO DAGScheduler - Missing parents: List()
17:38:08.761 INFO DAGScheduler - Submitting ResultStage 178 (MapPartitionsRDD[821] at filter at ReadsSparkSource.java:96), which has no missing parents
17:38:08.776 INFO MemoryStore - Block broadcast_349 stored as values in memory (estimated size 426.1 KiB, free 1918.7 MiB)
17:38:08.777 INFO MemoryStore - Block broadcast_349_piece0 stored as bytes in memory (estimated size 153.6 KiB, free 1918.5 MiB)
17:38:08.778 INFO BlockManagerInfo - Added broadcast_349_piece0 in memory on localhost:37953 (size: 153.6 KiB, free: 1919.7 MiB)
17:38:08.778 INFO SparkContext - Created broadcast 349 from broadcast at DAGScheduler.scala:1580
17:38:08.778 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 178 (MapPartitionsRDD[821] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:38:08.778 INFO TaskSchedulerImpl - Adding task set 178.0 with 1 tasks resource profile 0
17:38:08.779 INFO TaskSetManager - Starting task 0.0 in stage 178.0 (TID 234) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7872 bytes)
17:38:08.779 INFO Executor - Running task 0.0 in stage 178.0 (TID 234)
17:38:08.809 INFO NewHadoopRDD - Input split: file:/home/runner/work/gatk/gatk/src/test/resources/org/broadinstitute/hellbender/tools/BQSR/HiSeq.1mb.1RG.2k_lines.bam:0+222075
17:38:08.810 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:08.811 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:08.818 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:08.822 INFO Executor - Finished task 0.0 in stage 178.0 (TID 234). 989 bytes result sent to driver
17:38:08.822 INFO TaskSetManager - Finished task 0.0 in stage 178.0 (TID 234) in 44 ms on localhost (executor driver) (1/1)
17:38:08.822 INFO TaskSchedulerImpl - Removed TaskSet 178.0, whose tasks have all completed, from pool
17:38:08.822 INFO DAGScheduler - ResultStage 178 (count at ReadsSparkSinkUnitTest.java:185) finished in 0.061 s
17:38:08.822 INFO DAGScheduler - Job 131 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:08.822 INFO TaskSchedulerImpl - Killing all running tasks in stage 178: Stage finished
17:38:08.822 INFO DAGScheduler - Job 131 finished: count at ReadsSparkSinkUnitTest.java:185, took 0.061688 s
17:38:08.825 INFO SparkContext - Starting job: count at ReadsSparkSinkUnitTest.java:185
17:38:08.826 INFO DAGScheduler - Got job 132 (count at ReadsSparkSinkUnitTest.java:185) with 1 output partitions
17:38:08.826 INFO DAGScheduler - Final stage: ResultStage 179 (count at ReadsSparkSinkUnitTest.java:185)
17:38:08.826 INFO DAGScheduler - Parents of final stage: List()
17:38:08.826 INFO DAGScheduler - Missing parents: List()
17:38:08.826 INFO DAGScheduler - Submitting ResultStage 179 (MapPartitionsRDD[839] at filter at ReadsSparkSource.java:96), which has no missing parents
17:38:08.831 INFO MemoryStore - Block broadcast_350 stored as values in memory (estimated size 148.1 KiB, free 1918.4 MiB)
17:38:08.832 INFO MemoryStore - Block broadcast_350_piece0 stored as bytes in memory (estimated size 54.5 KiB, free 1918.3 MiB)
17:38:08.832 INFO BlockManagerInfo - Added broadcast_350_piece0 in memory on localhost:37953 (size: 54.5 KiB, free: 1919.6 MiB)
17:38:08.832 INFO SparkContext - Created broadcast 350 from broadcast at DAGScheduler.scala:1580
17:38:08.832 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 179 (MapPartitionsRDD[839] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:38:08.833 INFO TaskSchedulerImpl - Adding task set 179.0 with 1 tasks resource profile 0
17:38:08.833 INFO TaskSetManager - Starting task 0.0 in stage 179.0 (TID 235) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7810 bytes)
17:38:08.833 INFO Executor - Running task 0.0 in stage 179.0 (TID 235)
17:38:08.844 INFO NewHadoopRDD - Input split: file:/tmp/ReadsSparkSinkUnitTest112110568822137566832.bam:0+238969
17:38:08.845 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:08.848 INFO Executor - Finished task 0.0 in stage 179.0 (TID 235). 989 bytes result sent to driver
17:38:08.849 INFO TaskSetManager - Finished task 0.0 in stage 179.0 (TID 235) in 16 ms on localhost (executor driver) (1/1)
17:38:08.849 INFO TaskSchedulerImpl - Removed TaskSet 179.0, whose tasks have all completed, from pool
17:38:08.849 INFO DAGScheduler - ResultStage 179 (count at ReadsSparkSinkUnitTest.java:185) finished in 0.023 s
17:38:08.849 INFO DAGScheduler - Job 132 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:08.849 INFO TaskSchedulerImpl - Killing all running tasks in stage 179: Stage finished
17:38:08.849 INFO DAGScheduler - Job 132 finished: count at ReadsSparkSinkUnitTest.java:185, took 0.023458 s
17:38:08.854 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:08.856 INFO MemoryStore - Block broadcast_351 stored as values in memory (estimated size 297.9 KiB, free 1918.0 MiB)
17:38:08.861 INFO MemoryStore - Block broadcast_351_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1918.0 MiB)
17:38:08.861 INFO BlockManagerInfo - Added broadcast_351_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.6 MiB)
17:38:08.861 INFO SparkContext - Created broadcast 351 from newAPIHadoopFile at PathSplitSource.java:96
17:38:08.880 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:08.882 INFO MemoryStore - Block broadcast_352 stored as values in memory (estimated size 297.9 KiB, free 1917.7 MiB)
17:38:08.886 INFO BlockManagerInfo - Removed broadcast_340_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.6 MiB)
17:38:08.887 INFO BlockManagerInfo - Removed broadcast_350_piece0 on localhost:37953 in memory (size: 54.5 KiB, free: 1919.7 MiB)
17:38:08.888 INFO BlockManagerInfo - Removed broadcast_348_piece0 on localhost:37953 in memory (size: 54.6 KiB, free: 1919.7 MiB)
17:38:08.888 INFO BlockManagerInfo - Removed broadcast_349_piece0 on localhost:37953 in memory (size: 153.6 KiB, free: 1919.9 MiB)
17:38:08.889 INFO BlockManagerInfo - Removed broadcast_347_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.9 MiB)
17:38:08.890 INFO BlockManagerInfo - Removed broadcast_346_piece0 on localhost:37953 in memory (size: 8.3 KiB, free: 1920.0 MiB)
17:38:08.890 INFO MemoryStore - Block broadcast_352_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1919.3 MiB)
17:38:08.890 INFO BlockManagerInfo - Added broadcast_352_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.9 MiB)
17:38:08.890 INFO SparkContext - Created broadcast 352 from newAPIHadoopFile at PathSplitSource.java:96
17:38:08.909 INFO FileInputFormat - Total input files to process : 1
17:38:08.911 INFO MemoryStore - Block broadcast_353 stored as values in memory (estimated size 160.7 KiB, free 1919.2 MiB)
17:38:08.911 INFO MemoryStore - Block broadcast_353_piece0 stored as bytes in memory (estimated size 9.6 KiB, free 1919.2 MiB)
17:38:08.911 INFO BlockManagerInfo - Added broadcast_353_piece0 in memory on localhost:37953 (size: 9.6 KiB, free: 1919.9 MiB)
17:38:08.912 INFO SparkContext - Created broadcast 353 from broadcast at ReadsSparkSink.java:133
17:38:08.912 WARN HtsjdkReadsRddStorage - Unrecognized write option: DISABLE
17:38:08.913 INFO MemoryStore - Block broadcast_354 stored as values in memory (estimated size 163.2 KiB, free 1919.0 MiB)
17:38:08.913 INFO MemoryStore - Block broadcast_354_piece0 stored as bytes in memory (estimated size 9.6 KiB, free 1919.0 MiB)
17:38:08.914 INFO BlockManagerInfo - Added broadcast_354_piece0 in memory on localhost:37953 (size: 9.6 KiB, free: 1919.9 MiB)
17:38:08.914 INFO SparkContext - Created broadcast 354 from broadcast at BamSink.java:76
17:38:08.916 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:08.916 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:08.916 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:08.928 INFO SparkContext - Starting job: runJob at SparkHadoopWriter.scala:83
17:38:08.929 INFO DAGScheduler - Registering RDD 853 (mapToPair at SparkUtils.java:161) as input to shuffle 36
17:38:08.929 INFO DAGScheduler - Got job 133 (runJob at SparkHadoopWriter.scala:83) with 1 output partitions
17:38:08.929 INFO DAGScheduler - Final stage: ResultStage 181 (runJob at SparkHadoopWriter.scala:83)
17:38:08.929 INFO DAGScheduler - Parents of final stage: List(ShuffleMapStage 180)
17:38:08.929 INFO DAGScheduler - Missing parents: List(ShuffleMapStage 180)
17:38:08.929 INFO DAGScheduler - Submitting ShuffleMapStage 180 (MapPartitionsRDD[853] at mapToPair at SparkUtils.java:161), which has no missing parents
17:38:08.945 INFO MemoryStore - Block broadcast_355 stored as values in memory (estimated size 520.4 KiB, free 1918.5 MiB)
17:38:08.946 INFO MemoryStore - Block broadcast_355_piece0 stored as bytes in memory (estimated size 166.1 KiB, free 1918.3 MiB)
17:38:08.946 INFO BlockManagerInfo - Added broadcast_355_piece0 in memory on localhost:37953 (size: 166.1 KiB, free: 1919.7 MiB)
17:38:08.946 INFO SparkContext - Created broadcast 355 from broadcast at DAGScheduler.scala:1580
17:38:08.946 INFO DAGScheduler - Submitting 1 missing tasks from ShuffleMapStage 180 (MapPartitionsRDD[853] at mapToPair at SparkUtils.java:161) (first 15 tasks are for partitions Vector(0))
17:38:08.946 INFO TaskSchedulerImpl - Adding task set 180.0 with 1 tasks resource profile 0
17:38:08.947 INFO TaskSetManager - Starting task 0.0 in stage 180.0 (TID 236) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7861 bytes)
17:38:08.947 INFO Executor - Running task 0.0 in stage 180.0 (TID 236)
17:38:08.979 INFO NewHadoopRDD - Input split: file:/home/runner/work/gatk/gatk/src/test/resources/org/broadinstitute/hellbender/tools/BQSR/HiSeq.1mb.1RG.2k_lines.bam:0+222075
17:38:08.980 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:08.981 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:08.988 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:08.996 INFO Executor - Finished task 0.0 in stage 180.0 (TID 236). 1148 bytes result sent to driver
17:38:08.997 INFO TaskSetManager - Finished task 0.0 in stage 180.0 (TID 236) in 50 ms on localhost (executor driver) (1/1)
17:38:08.997 INFO TaskSchedulerImpl - Removed TaskSet 180.0, whose tasks have all completed, from pool
17:38:08.997 INFO DAGScheduler - ShuffleMapStage 180 (mapToPair at SparkUtils.java:161) finished in 0.068 s
17:38:08.997 INFO DAGScheduler - looking for newly runnable stages
17:38:08.997 INFO DAGScheduler - running: HashSet()
17:38:08.997 INFO DAGScheduler - waiting: HashSet(ResultStage 181)
17:38:08.997 INFO DAGScheduler - failed: HashSet()
17:38:08.997 INFO DAGScheduler - Submitting ResultStage 181 (MapPartitionsRDD[858] at mapToPair at BamSink.java:91), which has no missing parents
17:38:09.003 INFO MemoryStore - Block broadcast_356 stored as values in memory (estimated size 241.4 KiB, free 1918.1 MiB)
17:38:09.004 INFO MemoryStore - Block broadcast_356_piece0 stored as bytes in memory (estimated size 67.0 KiB, free 1918.0 MiB)
17:38:09.004 INFO BlockManagerInfo - Added broadcast_356_piece0 in memory on localhost:37953 (size: 67.0 KiB, free: 1919.7 MiB)
17:38:09.004 INFO SparkContext - Created broadcast 356 from broadcast at DAGScheduler.scala:1580
17:38:09.004 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 181 (MapPartitionsRDD[858] at mapToPair at BamSink.java:91) (first 15 tasks are for partitions Vector(0))
17:38:09.004 INFO TaskSchedulerImpl - Adding task set 181.0 with 1 tasks resource profile 0
17:38:09.005 INFO TaskSetManager - Starting task 0.0 in stage 181.0 (TID 237) (localhost, executor driver, partition 0, NODE_LOCAL, 7513 bytes)
17:38:09.005 INFO Executor - Running task 0.0 in stage 181.0 (TID 237)
17:38:09.010 INFO ShuffleBlockFetcherIterator - Getting 1 (343.8 KiB) non-empty blocks including 1 (343.8 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
17:38:09.010 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
17:38:09.021 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:09.022 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:09.022 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:09.022 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:09.022 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:09.022 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:09.029 WARN IntelDeflaterFactory - Intel Deflater not supported, using Java.util.zip.Deflater
17:38:09.040 INFO FileOutputCommitter - Saved output of task 'attempt_20251027173808740282814132400956_0858_r_000000_0' to file:/tmp/ReadsSparkSinkUnitTest1.someOtherPlace3972171325811898217/_temporary/0/task_20251027173808740282814132400956_0858_r_000000
17:38:09.040 INFO SparkHadoopMapRedUtil - attempt_20251027173808740282814132400956_0858_r_000000_0: Committed. Elapsed time: 0 ms.
17:38:09.041 INFO Executor - Finished task 0.0 in stage 181.0 (TID 237). 1858 bytes result sent to driver
17:38:09.041 INFO TaskSetManager - Finished task 0.0 in stage 181.0 (TID 237) in 36 ms on localhost (executor driver) (1/1)
17:38:09.041 INFO TaskSchedulerImpl - Removed TaskSet 181.0, whose tasks have all completed, from pool
17:38:09.041 INFO DAGScheduler - ResultStage 181 (runJob at SparkHadoopWriter.scala:83) finished in 0.044 s
17:38:09.041 INFO DAGScheduler - Job 133 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:09.041 INFO TaskSchedulerImpl - Killing all running tasks in stage 181: Stage finished
17:38:09.041 INFO DAGScheduler - Job 133 finished: runJob at SparkHadoopWriter.scala:83, took 0.113174 s
17:38:09.042 INFO SparkHadoopWriter - Start to commit write Job job_20251027173808740282814132400956_0858.
17:38:09.045 INFO SparkHadoopWriter - Write Job job_20251027173808740282814132400956_0858 committed. Elapsed time: 3 ms.
17:38:09.047 WARN IntelDeflaterFactory - Intel Deflater not supported, using Java.util.zip.Deflater
17:38:09.053 INFO HadoopFileSystemWrapper - Concatenating 3 parts to /tmp/ReadsSparkSinkUnitTest16237101560704753700.bam
17:38:09.056 INFO HadoopFileSystemWrapper - Concatenating to /tmp/ReadsSparkSinkUnitTest16237101560704753700.bam done
17:38:09.056 INFO IndexFileMerger - Merging .bai files in temp directory /tmp/ReadsSparkSinkUnitTest1.someOtherPlace3972171325811898217 to /tmp/ReadsSparkSinkUnitTest16237101560704753700.bam.bai
17:38:09.060 INFO IndexFileMerger - Done merging .bai files
17:38:09.060 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:09.062 INFO MemoryStore - Block broadcast_357 stored as values in memory (estimated size 297.9 KiB, free 1917.7 MiB)
17:38:09.068 INFO MemoryStore - Block broadcast_357_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1917.7 MiB)
17:38:09.068 INFO BlockManagerInfo - Added broadcast_357_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.6 MiB)
17:38:09.068 INFO SparkContext - Created broadcast 357 from newAPIHadoopFile at PathSplitSource.java:96
17:38:09.087 INFO FileInputFormat - Total input files to process : 1
17:38:09.119 INFO SparkContext - Starting job: collect at ReadsSparkSinkUnitTest.java:182
17:38:09.119 INFO DAGScheduler - Got job 134 (collect at ReadsSparkSinkUnitTest.java:182) with 1 output partitions
17:38:09.119 INFO DAGScheduler - Final stage: ResultStage 182 (collect at ReadsSparkSinkUnitTest.java:182)
17:38:09.119 INFO DAGScheduler - Parents of final stage: List()
17:38:09.119 INFO DAGScheduler - Missing parents: List()
17:38:09.119 INFO DAGScheduler - Submitting ResultStage 182 (MapPartitionsRDD[865] at filter at ReadsSparkSource.java:96), which has no missing parents
17:38:09.135 INFO MemoryStore - Block broadcast_358 stored as values in memory (estimated size 426.2 KiB, free 1917.3 MiB)
17:38:09.136 INFO MemoryStore - Block broadcast_358_piece0 stored as bytes in memory (estimated size 153.6 KiB, free 1917.1 MiB)
17:38:09.136 INFO BlockManagerInfo - Added broadcast_358_piece0 in memory on localhost:37953 (size: 153.6 KiB, free: 1919.5 MiB)
17:38:09.136 INFO SparkContext - Created broadcast 358 from broadcast at DAGScheduler.scala:1580
17:38:09.136 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 182 (MapPartitionsRDD[865] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:38:09.136 INFO TaskSchedulerImpl - Adding task set 182.0 with 1 tasks resource profile 0
17:38:09.137 INFO TaskSetManager - Starting task 0.0 in stage 182.0 (TID 238) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7809 bytes)
17:38:09.137 INFO Executor - Running task 0.0 in stage 182.0 (TID 238)
17:38:09.167 INFO NewHadoopRDD - Input split: file:/tmp/ReadsSparkSinkUnitTest16237101560704753700.bam:0+238969
17:38:09.168 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:09.169 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:09.178 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:09.185 INFO BlockManagerInfo - Removed broadcast_356_piece0 on localhost:37953 in memory (size: 67.0 KiB, free: 1919.5 MiB)
17:38:09.186 INFO Executor - Finished task 0.0 in stage 182.0 (TID 238). 651569 bytes result sent to driver
17:38:09.186 INFO BlockManagerInfo - Removed broadcast_355_piece0 on localhost:37953 in memory (size: 166.1 KiB, free: 1919.7 MiB)
17:38:09.187 INFO BlockManagerInfo - Removed broadcast_352_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.7 MiB)
17:38:09.187 INFO TaskSetManager - Finished task 0.0 in stage 182.0 (TID 238) in 50 ms on localhost (executor driver) (1/1)
17:38:09.187 INFO TaskSchedulerImpl - Removed TaskSet 182.0, whose tasks have all completed, from pool
17:38:09.188 INFO DAGScheduler - ResultStage 182 (collect at ReadsSparkSinkUnitTest.java:182) finished in 0.068 s
17:38:09.188 INFO DAGScheduler - Job 134 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:09.188 INFO TaskSchedulerImpl - Killing all running tasks in stage 182: Stage finished
17:38:09.188 INFO DAGScheduler - Job 134 finished: collect at ReadsSparkSinkUnitTest.java:182, took 0.068956 s
17:38:09.188 INFO BlockManagerInfo - Removed broadcast_353_piece0 on localhost:37953 in memory (size: 9.6 KiB, free: 1919.7 MiB)
17:38:09.189 INFO BlockManagerInfo - Removed broadcast_354_piece0 on localhost:37953 in memory (size: 9.6 KiB, free: 1919.8 MiB)
17:38:09.198 INFO SparkContext - Starting job: count at ReadsSparkSinkUnitTest.java:185
17:38:09.198 INFO DAGScheduler - Got job 135 (count at ReadsSparkSinkUnitTest.java:185) with 1 output partitions
17:38:09.198 INFO DAGScheduler - Final stage: ResultStage 183 (count at ReadsSparkSinkUnitTest.java:185)
17:38:09.198 INFO DAGScheduler - Parents of final stage: List()
17:38:09.198 INFO DAGScheduler - Missing parents: List()
17:38:09.198 INFO DAGScheduler - Submitting ResultStage 183 (MapPartitionsRDD[846] at filter at ReadsSparkSource.java:96), which has no missing parents
17:38:09.213 INFO MemoryStore - Block broadcast_359 stored as values in memory (estimated size 426.1 KiB, free 1918.3 MiB)
17:38:09.214 INFO MemoryStore - Block broadcast_359_piece0 stored as bytes in memory (estimated size 153.6 KiB, free 1918.2 MiB)
17:38:09.214 INFO BlockManagerInfo - Added broadcast_359_piece0 in memory on localhost:37953 (size: 153.6 KiB, free: 1919.6 MiB)
17:38:09.215 INFO SparkContext - Created broadcast 359 from broadcast at DAGScheduler.scala:1580
17:38:09.215 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 183 (MapPartitionsRDD[846] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:38:09.215 INFO TaskSchedulerImpl - Adding task set 183.0 with 1 tasks resource profile 0
17:38:09.215 INFO TaskSetManager - Starting task 0.0 in stage 183.0 (TID 239) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7872 bytes)
17:38:09.216 INFO Executor - Running task 0.0 in stage 183.0 (TID 239)
17:38:09.246 INFO NewHadoopRDD - Input split: file:/home/runner/work/gatk/gatk/src/test/resources/org/broadinstitute/hellbender/tools/BQSR/HiSeq.1mb.1RG.2k_lines.bam:0+222075
17:38:09.247 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:09.248 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:09.255 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:09.258 INFO Executor - Finished task 0.0 in stage 183.0 (TID 239). 989 bytes result sent to driver
17:38:09.259 INFO TaskSetManager - Finished task 0.0 in stage 183.0 (TID 239) in 44 ms on localhost (executor driver) (1/1)
17:38:09.259 INFO TaskSchedulerImpl - Removed TaskSet 183.0, whose tasks have all completed, from pool
17:38:09.259 INFO DAGScheduler - ResultStage 183 (count at ReadsSparkSinkUnitTest.java:185) finished in 0.061 s
17:38:09.259 INFO DAGScheduler - Job 135 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:09.259 INFO TaskSchedulerImpl - Killing all running tasks in stage 183: Stage finished
17:38:09.259 INFO DAGScheduler - Job 135 finished: count at ReadsSparkSinkUnitTest.java:185, took 0.061392 s
17:38:09.262 INFO SparkContext - Starting job: count at ReadsSparkSinkUnitTest.java:185
17:38:09.262 INFO DAGScheduler - Got job 136 (count at ReadsSparkSinkUnitTest.java:185) with 1 output partitions
17:38:09.262 INFO DAGScheduler - Final stage: ResultStage 184 (count at ReadsSparkSinkUnitTest.java:185)
17:38:09.262 INFO DAGScheduler - Parents of final stage: List()
17:38:09.262 INFO DAGScheduler - Missing parents: List()
17:38:09.262 INFO DAGScheduler - Submitting ResultStage 184 (MapPartitionsRDD[865] at filter at ReadsSparkSource.java:96), which has no missing parents
17:38:09.278 INFO MemoryStore - Block broadcast_360 stored as values in memory (estimated size 426.1 KiB, free 1917.8 MiB)
17:38:09.279 INFO MemoryStore - Block broadcast_360_piece0 stored as bytes in memory (estimated size 153.6 KiB, free 1917.6 MiB)
17:38:09.279 INFO BlockManagerInfo - Added broadcast_360_piece0 in memory on localhost:37953 (size: 153.6 KiB, free: 1919.5 MiB)
17:38:09.279 INFO SparkContext - Created broadcast 360 from broadcast at DAGScheduler.scala:1580
17:38:09.279 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 184 (MapPartitionsRDD[865] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:38:09.279 INFO TaskSchedulerImpl - Adding task set 184.0 with 1 tasks resource profile 0
17:38:09.280 INFO TaskSetManager - Starting task 0.0 in stage 184.0 (TID 240) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7809 bytes)
17:38:09.280 INFO Executor - Running task 0.0 in stage 184.0 (TID 240)
17:38:09.310 INFO NewHadoopRDD - Input split: file:/tmp/ReadsSparkSinkUnitTest16237101560704753700.bam:0+238969
17:38:09.311 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:09.312 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:09.321 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:09.325 INFO Executor - Finished task 0.0 in stage 184.0 (TID 240). 989 bytes result sent to driver
17:38:09.325 INFO TaskSetManager - Finished task 0.0 in stage 184.0 (TID 240) in 46 ms on localhost (executor driver) (1/1)
17:38:09.326 INFO TaskSchedulerImpl - Removed TaskSet 184.0, whose tasks have all completed, from pool
17:38:09.326 INFO DAGScheduler - ResultStage 184 (count at ReadsSparkSinkUnitTest.java:185) finished in 0.063 s
17:38:09.326 INFO DAGScheduler - Job 136 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:09.326 INFO TaskSchedulerImpl - Killing all running tasks in stage 184: Stage finished
17:38:09.326 INFO DAGScheduler - Job 136 finished: count at ReadsSparkSinkUnitTest.java:185, took 0.063834 s
17:38:09.330 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:09.333 INFO MemoryStore - Block broadcast_361 stored as values in memory (estimated size 297.9 KiB, free 1917.3 MiB)
17:38:09.338 INFO MemoryStore - Block broadcast_361_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1917.3 MiB)
17:38:09.338 INFO BlockManagerInfo - Added broadcast_361_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.4 MiB)
17:38:09.338 INFO SparkContext - Created broadcast 361 from newAPIHadoopFile at PathSplitSource.java:96
17:38:09.357 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:09.359 INFO MemoryStore - Block broadcast_362 stored as values in memory (estimated size 297.9 KiB, free 1917.0 MiB)
17:38:09.365 INFO MemoryStore - Block broadcast_362_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1916.9 MiB)
17:38:09.365 INFO BlockManagerInfo - Added broadcast_362_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.4 MiB)
17:38:09.365 INFO SparkContext - Created broadcast 362 from newAPIHadoopFile at PathSplitSource.java:96
17:38:09.384 INFO FileInputFormat - Total input files to process : 1
17:38:09.386 INFO MemoryStore - Block broadcast_363 stored as values in memory (estimated size 160.7 KiB, free 1916.8 MiB)
17:38:09.386 INFO MemoryStore - Block broadcast_363_piece0 stored as bytes in memory (estimated size 9.6 KiB, free 1916.8 MiB)
17:38:09.386 INFO BlockManagerInfo - Added broadcast_363_piece0 in memory on localhost:37953 (size: 9.6 KiB, free: 1919.3 MiB)
17:38:09.387 INFO SparkContext - Created broadcast 363 from broadcast at ReadsSparkSink.java:133
17:38:09.387 WARN HtsjdkReadsRddStorage - Unrecognized write option: DISABLE
17:38:09.388 INFO MemoryStore - Block broadcast_364 stored as values in memory (estimated size 163.2 KiB, free 1916.6 MiB)
17:38:09.389 INFO MemoryStore - Block broadcast_364_piece0 stored as bytes in memory (estimated size 9.6 KiB, free 1916.6 MiB)
17:38:09.389 INFO BlockManagerInfo - Added broadcast_364_piece0 in memory on localhost:37953 (size: 9.6 KiB, free: 1919.3 MiB)
17:38:09.389 INFO SparkContext - Created broadcast 364 from broadcast at BamSink.java:76
17:38:09.391 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:09.391 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:09.391 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:09.403 INFO SparkContext - Starting job: runJob at SparkHadoopWriter.scala:83
17:38:09.404 INFO DAGScheduler - Registering RDD 879 (mapToPair at SparkUtils.java:161) as input to shuffle 37
17:38:09.404 INFO DAGScheduler - Got job 137 (runJob at SparkHadoopWriter.scala:83) with 1 output partitions
17:38:09.404 INFO DAGScheduler - Final stage: ResultStage 186 (runJob at SparkHadoopWriter.scala:83)
17:38:09.404 INFO DAGScheduler - Parents of final stage: List(ShuffleMapStage 185)
17:38:09.404 INFO DAGScheduler - Missing parents: List(ShuffleMapStage 185)
17:38:09.404 INFO DAGScheduler - Submitting ShuffleMapStage 185 (MapPartitionsRDD[879] at mapToPair at SparkUtils.java:161), which has no missing parents
17:38:09.420 INFO MemoryStore - Block broadcast_365 stored as values in memory (estimated size 520.4 KiB, free 1916.1 MiB)
17:38:09.421 INFO MemoryStore - Block broadcast_365_piece0 stored as bytes in memory (estimated size 166.1 KiB, free 1915.9 MiB)
17:38:09.421 INFO BlockManagerInfo - Added broadcast_365_piece0 in memory on localhost:37953 (size: 166.1 KiB, free: 1919.2 MiB)
17:38:09.422 INFO SparkContext - Created broadcast 365 from broadcast at DAGScheduler.scala:1580
17:38:09.422 INFO DAGScheduler - Submitting 1 missing tasks from ShuffleMapStage 185 (MapPartitionsRDD[879] at mapToPair at SparkUtils.java:161) (first 15 tasks are for partitions Vector(0))
17:38:09.422 INFO TaskSchedulerImpl - Adding task set 185.0 with 1 tasks resource profile 0
17:38:09.422 INFO TaskSetManager - Starting task 0.0 in stage 185.0 (TID 241) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7861 bytes)
17:38:09.423 INFO Executor - Running task 0.0 in stage 185.0 (TID 241)
17:38:09.454 INFO NewHadoopRDD - Input split: file:/home/runner/work/gatk/gatk/src/test/resources/org/broadinstitute/hellbender/tools/BQSR/HiSeq.1mb.1RG.2k_lines.bam:0+222075
17:38:09.455 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:09.456 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:09.463 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:09.474 INFO Executor - Finished task 0.0 in stage 185.0 (TID 241). 1234 bytes result sent to driver
17:38:09.475 INFO TaskSetManager - Finished task 0.0 in stage 185.0 (TID 241) in 53 ms on localhost (executor driver) (1/1)
17:38:09.475 INFO TaskSchedulerImpl - Removed TaskSet 185.0, whose tasks have all completed, from pool
17:38:09.475 INFO DAGScheduler - ShuffleMapStage 185 (mapToPair at SparkUtils.java:161) finished in 0.071 s
17:38:09.475 INFO DAGScheduler - looking for newly runnable stages
17:38:09.475 INFO DAGScheduler - running: HashSet()
17:38:09.475 INFO DAGScheduler - waiting: HashSet(ResultStage 186)
17:38:09.475 INFO DAGScheduler - failed: HashSet()
17:38:09.475 INFO DAGScheduler - Submitting ResultStage 186 (MapPartitionsRDD[884] at mapToPair at BamSink.java:91), which has no missing parents
17:38:09.475 INFO BlockManagerInfo - Removed broadcast_360_piece0 on localhost:37953 in memory (size: 153.6 KiB, free: 1919.3 MiB)
17:38:09.476 INFO BlockManagerInfo - Removed broadcast_359_piece0 on localhost:37953 in memory (size: 153.6 KiB, free: 1919.5 MiB)
17:38:09.477 INFO BlockManagerInfo - Removed broadcast_362_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.5 MiB)
17:38:09.477 INFO BlockManagerInfo - Removed broadcast_351_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.6 MiB)
17:38:09.478 INFO BlockManagerInfo - Removed broadcast_357_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.6 MiB)
17:38:09.478 INFO BlockManagerInfo - Removed broadcast_358_piece0 on localhost:37953 in memory (size: 153.6 KiB, free: 1919.8 MiB)
17:38:09.481 INFO MemoryStore - Block broadcast_366 stored as values in memory (estimated size 241.4 KiB, free 1918.4 MiB)
17:38:09.482 INFO MemoryStore - Block broadcast_366_piece0 stored as bytes in memory (estimated size 67.0 KiB, free 1918.4 MiB)
17:38:09.482 INFO BlockManagerInfo - Added broadcast_366_piece0 in memory on localhost:37953 (size: 67.0 KiB, free: 1919.7 MiB)
17:38:09.482 INFO SparkContext - Created broadcast 366 from broadcast at DAGScheduler.scala:1580
17:38:09.482 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 186 (MapPartitionsRDD[884] at mapToPair at BamSink.java:91) (first 15 tasks are for partitions Vector(0))
17:38:09.482 INFO TaskSchedulerImpl - Adding task set 186.0 with 1 tasks resource profile 0
17:38:09.483 INFO TaskSetManager - Starting task 0.0 in stage 186.0 (TID 242) (localhost, executor driver, partition 0, NODE_LOCAL, 7513 bytes)
17:38:09.483 INFO Executor - Running task 0.0 in stage 186.0 (TID 242)
17:38:09.488 INFO ShuffleBlockFetcherIterator - Getting 1 (343.8 KiB) non-empty blocks including 1 (343.8 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
17:38:09.488 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
17:38:09.499 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:09.499 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:09.499 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:09.499 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:09.499 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:09.499 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:09.506 WARN IntelDeflaterFactory - Intel Deflater not supported, using Java.util.zip.Deflater
17:38:09.520 INFO FileOutputCommitter - Saved output of task 'attempt_202510271738092696567410680979701_0884_r_000000_0' to file:/tmp/ReadsSparkSinkUnitTest1.someOtherPlace7583345549138248864/_temporary/0/task_202510271738092696567410680979701_0884_r_000000
17:38:09.520 INFO SparkHadoopMapRedUtil - attempt_202510271738092696567410680979701_0884_r_000000_0: Committed. Elapsed time: 0 ms.
17:38:09.520 INFO Executor - Finished task 0.0 in stage 186.0 (TID 242). 1858 bytes result sent to driver
17:38:09.520 INFO TaskSetManager - Finished task 0.0 in stage 186.0 (TID 242) in 37 ms on localhost (executor driver) (1/1)
17:38:09.520 INFO TaskSchedulerImpl - Removed TaskSet 186.0, whose tasks have all completed, from pool
17:38:09.521 INFO DAGScheduler - ResultStage 186 (runJob at SparkHadoopWriter.scala:83) finished in 0.045 s
17:38:09.521 INFO DAGScheduler - Job 137 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:09.521 INFO TaskSchedulerImpl - Killing all running tasks in stage 186: Stage finished
17:38:09.521 INFO DAGScheduler - Job 137 finished: runJob at SparkHadoopWriter.scala:83, took 0.117237 s
17:38:09.521 INFO SparkHadoopWriter - Start to commit write Job job_202510271738092696567410680979701_0884.
17:38:09.524 INFO SparkHadoopWriter - Write Job job_202510271738092696567410680979701_0884 committed. Elapsed time: 3 ms.
17:38:09.527 WARN IntelDeflaterFactory - Intel Deflater not supported, using Java.util.zip.Deflater
17:38:09.532 INFO HadoopFileSystemWrapper - Concatenating 3 parts to /tmp/ReadsSparkSinkUnitTest112357376929615774168.bam
17:38:09.535 INFO HadoopFileSystemWrapper - Concatenating to /tmp/ReadsSparkSinkUnitTest112357376929615774168.bam done
17:38:09.535 INFO IndexFileMerger - Merging .sbi files in temp directory /tmp/ReadsSparkSinkUnitTest1.someOtherPlace7583345549138248864 to /tmp/ReadsSparkSinkUnitTest112357376929615774168.bam.sbi
17:38:09.539 INFO IndexFileMerger - Done merging .sbi files
17:38:09.540 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:09.541 INFO MemoryStore - Block broadcast_367 stored as values in memory (estimated size 320.0 B, free 1918.4 MiB)
17:38:09.541 INFO MemoryStore - Block broadcast_367_piece0 stored as bytes in memory (estimated size 233.0 B, free 1918.4 MiB)
17:38:09.542 INFO BlockManagerInfo - Added broadcast_367_piece0 in memory on localhost:37953 (size: 233.0 B, free: 1919.7 MiB)
17:38:09.542 INFO SparkContext - Created broadcast 367 from broadcast at BamSource.java:104
17:38:09.543 INFO MemoryStore - Block broadcast_368 stored as values in memory (estimated size 297.9 KiB, free 1918.1 MiB)
17:38:09.548 INFO MemoryStore - Block broadcast_368_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1918.0 MiB)
17:38:09.548 INFO BlockManagerInfo - Added broadcast_368_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.7 MiB)
17:38:09.549 INFO SparkContext - Created broadcast 368 from newAPIHadoopFile at PathSplitSource.java:96
17:38:09.557 INFO FileInputFormat - Total input files to process : 1
17:38:09.570 INFO SparkContext - Starting job: collect at ReadsSparkSinkUnitTest.java:182
17:38:09.570 INFO DAGScheduler - Got job 138 (collect at ReadsSparkSinkUnitTest.java:182) with 1 output partitions
17:38:09.570 INFO DAGScheduler - Final stage: ResultStage 187 (collect at ReadsSparkSinkUnitTest.java:182)
17:38:09.570 INFO DAGScheduler - Parents of final stage: List()
17:38:09.570 INFO DAGScheduler - Missing parents: List()
17:38:09.570 INFO DAGScheduler - Submitting ResultStage 187 (MapPartitionsRDD[890] at filter at ReadsSparkSource.java:96), which has no missing parents
17:38:09.576 INFO MemoryStore - Block broadcast_369 stored as values in memory (estimated size 148.2 KiB, free 1917.9 MiB)
17:38:09.576 INFO MemoryStore - Block broadcast_369_piece0 stored as bytes in memory (estimated size 54.5 KiB, free 1917.8 MiB)
17:38:09.576 INFO BlockManagerInfo - Added broadcast_369_piece0 in memory on localhost:37953 (size: 54.5 KiB, free: 1919.6 MiB)
17:38:09.577 INFO SparkContext - Created broadcast 369 from broadcast at DAGScheduler.scala:1580
17:38:09.577 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 187 (MapPartitionsRDD[890] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:38:09.577 INFO TaskSchedulerImpl - Adding task set 187.0 with 1 tasks resource profile 0
17:38:09.577 INFO TaskSetManager - Starting task 0.0 in stage 187.0 (TID 243) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7810 bytes)
17:38:09.577 INFO Executor - Running task 0.0 in stage 187.0 (TID 243)
17:38:09.589 INFO NewHadoopRDD - Input split: file:/tmp/ReadsSparkSinkUnitTest112357376929615774168.bam:0+238969
17:38:09.590 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:09.594 INFO Executor - Finished task 0.0 in stage 187.0 (TID 243). 651483 bytes result sent to driver
17:38:09.596 INFO TaskSetManager - Finished task 0.0 in stage 187.0 (TID 243) in 19 ms on localhost (executor driver) (1/1)
17:38:09.596 INFO TaskSchedulerImpl - Removed TaskSet 187.0, whose tasks have all completed, from pool
17:38:09.596 INFO DAGScheduler - ResultStage 187 (collect at ReadsSparkSinkUnitTest.java:182) finished in 0.026 s
17:38:09.596 INFO DAGScheduler - Job 138 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:09.596 INFO TaskSchedulerImpl - Killing all running tasks in stage 187: Stage finished
17:38:09.596 INFO DAGScheduler - Job 138 finished: collect at ReadsSparkSinkUnitTest.java:182, took 0.026539 s
17:38:09.606 INFO SparkContext - Starting job: count at ReadsSparkSinkUnitTest.java:185
17:38:09.606 INFO DAGScheduler - Got job 139 (count at ReadsSparkSinkUnitTest.java:185) with 1 output partitions
17:38:09.606 INFO DAGScheduler - Final stage: ResultStage 188 (count at ReadsSparkSinkUnitTest.java:185)
17:38:09.606 INFO DAGScheduler - Parents of final stage: List()
17:38:09.606 INFO DAGScheduler - Missing parents: List()
17:38:09.606 INFO DAGScheduler - Submitting ResultStage 188 (MapPartitionsRDD[872] at filter at ReadsSparkSource.java:96), which has no missing parents
17:38:09.621 INFO MemoryStore - Block broadcast_370 stored as values in memory (estimated size 426.1 KiB, free 1917.4 MiB)
17:38:09.623 INFO MemoryStore - Block broadcast_370_piece0 stored as bytes in memory (estimated size 153.6 KiB, free 1917.2 MiB)
17:38:09.623 INFO BlockManagerInfo - Added broadcast_370_piece0 in memory on localhost:37953 (size: 153.6 KiB, free: 1919.5 MiB)
17:38:09.623 INFO SparkContext - Created broadcast 370 from broadcast at DAGScheduler.scala:1580
17:38:09.623 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 188 (MapPartitionsRDD[872] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:38:09.623 INFO TaskSchedulerImpl - Adding task set 188.0 with 1 tasks resource profile 0
17:38:09.624 INFO TaskSetManager - Starting task 0.0 in stage 188.0 (TID 244) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7872 bytes)
17:38:09.624 INFO Executor - Running task 0.0 in stage 188.0 (TID 244)
17:38:09.654 INFO NewHadoopRDD - Input split: file:/home/runner/work/gatk/gatk/src/test/resources/org/broadinstitute/hellbender/tools/BQSR/HiSeq.1mb.1RG.2k_lines.bam:0+222075
17:38:09.654 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:09.655 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:09.662 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:09.666 INFO Executor - Finished task 0.0 in stage 188.0 (TID 244). 989 bytes result sent to driver
17:38:09.666 INFO TaskSetManager - Finished task 0.0 in stage 188.0 (TID 244) in 43 ms on localhost (executor driver) (1/1)
17:38:09.666 INFO TaskSchedulerImpl - Removed TaskSet 188.0, whose tasks have all completed, from pool
17:38:09.666 INFO DAGScheduler - ResultStage 188 (count at ReadsSparkSinkUnitTest.java:185) finished in 0.059 s
17:38:09.666 INFO DAGScheduler - Job 139 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:09.666 INFO TaskSchedulerImpl - Killing all running tasks in stage 188: Stage finished
17:38:09.666 INFO DAGScheduler - Job 139 finished: count at ReadsSparkSinkUnitTest.java:185, took 0.060361 s
17:38:09.669 INFO SparkContext - Starting job: count at ReadsSparkSinkUnitTest.java:185
17:38:09.670 INFO DAGScheduler - Got job 140 (count at ReadsSparkSinkUnitTest.java:185) with 1 output partitions
17:38:09.670 INFO DAGScheduler - Final stage: ResultStage 189 (count at ReadsSparkSinkUnitTest.java:185)
17:38:09.670 INFO DAGScheduler - Parents of final stage: List()
17:38:09.670 INFO DAGScheduler - Missing parents: List()
17:38:09.670 INFO DAGScheduler - Submitting ResultStage 189 (MapPartitionsRDD[890] at filter at ReadsSparkSource.java:96), which has no missing parents
17:38:09.675 INFO MemoryStore - Block broadcast_371 stored as values in memory (estimated size 148.1 KiB, free 1917.1 MiB)
17:38:09.676 INFO MemoryStore - Block broadcast_371_piece0 stored as bytes in memory (estimated size 54.5 KiB, free 1917.1 MiB)
17:38:09.676 INFO BlockManagerInfo - Added broadcast_371_piece0 in memory on localhost:37953 (size: 54.5 KiB, free: 1919.4 MiB)
17:38:09.676 INFO SparkContext - Created broadcast 371 from broadcast at DAGScheduler.scala:1580
17:38:09.676 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 189 (MapPartitionsRDD[890] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:38:09.676 INFO TaskSchedulerImpl - Adding task set 189.0 with 1 tasks resource profile 0
17:38:09.677 INFO TaskSetManager - Starting task 0.0 in stage 189.0 (TID 245) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7810 bytes)
17:38:09.677 INFO Executor - Running task 0.0 in stage 189.0 (TID 245)
17:38:09.688 INFO NewHadoopRDD - Input split: file:/tmp/ReadsSparkSinkUnitTest112357376929615774168.bam:0+238969
17:38:09.688 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:09.696 INFO Executor - Finished task 0.0 in stage 189.0 (TID 245). 1075 bytes result sent to driver
17:38:09.696 INFO BlockManagerInfo - Removed broadcast_369_piece0 on localhost:37953 in memory (size: 54.5 KiB, free: 1919.5 MiB)
17:38:09.696 INFO TaskSetManager - Finished task 0.0 in stage 189.0 (TID 245) in 19 ms on localhost (executor driver) (1/1)
17:38:09.696 INFO TaskSchedulerImpl - Removed TaskSet 189.0, whose tasks have all completed, from pool
17:38:09.696 INFO DAGScheduler - ResultStage 189 (count at ReadsSparkSinkUnitTest.java:185) finished in 0.026 s
17:38:09.696 INFO DAGScheduler - Job 140 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:09.697 INFO TaskSchedulerImpl - Killing all running tasks in stage 189: Stage finished
17:38:09.697 INFO DAGScheduler - Job 140 finished: count at ReadsSparkSinkUnitTest.java:185, took 0.027214 s
17:38:09.697 INFO BlockManagerInfo - Removed broadcast_364_piece0 on localhost:37953 in memory (size: 9.6 KiB, free: 1919.5 MiB)
17:38:09.698 INFO BlockManagerInfo - Removed broadcast_370_piece0 on localhost:37953 in memory (size: 153.6 KiB, free: 1919.6 MiB)
17:38:09.698 INFO BlockManagerInfo - Removed broadcast_366_piece0 on localhost:37953 in memory (size: 67.0 KiB, free: 1919.7 MiB)
17:38:09.699 INFO BlockManagerInfo - Removed broadcast_365_piece0 on localhost:37953 in memory (size: 166.1 KiB, free: 1919.8 MiB)
17:38:09.699 INFO BlockManagerInfo - Removed broadcast_363_piece0 on localhost:37953 in memory (size: 9.6 KiB, free: 1919.8 MiB)
17:38:09.701 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:09.703 INFO MemoryStore - Block broadcast_372 stored as values in memory (estimated size 297.9 KiB, free 1918.8 MiB)
17:38:09.709 INFO MemoryStore - Block broadcast_372_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1918.8 MiB)
17:38:09.709 INFO BlockManagerInfo - Added broadcast_372_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.8 MiB)
17:38:09.709 INFO SparkContext - Created broadcast 372 from newAPIHadoopFile at PathSplitSource.java:96
17:38:09.728 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:09.730 INFO MemoryStore - Block broadcast_373 stored as values in memory (estimated size 297.9 KiB, free 1918.5 MiB)
17:38:09.735 INFO MemoryStore - Block broadcast_373_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1918.4 MiB)
17:38:09.735 INFO BlockManagerInfo - Added broadcast_373_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.8 MiB)
17:38:09.736 INFO SparkContext - Created broadcast 373 from newAPIHadoopFile at PathSplitSource.java:96
17:38:09.754 INFO FileInputFormat - Total input files to process : 1
17:38:09.756 INFO MemoryStore - Block broadcast_374 stored as values in memory (estimated size 160.7 KiB, free 1918.3 MiB)
17:38:09.757 INFO MemoryStore - Block broadcast_374_piece0 stored as bytes in memory (estimated size 9.6 KiB, free 1918.3 MiB)
17:38:09.757 INFO BlockManagerInfo - Added broadcast_374_piece0 in memory on localhost:37953 (size: 9.6 KiB, free: 1919.7 MiB)
17:38:09.757 INFO SparkContext - Created broadcast 374 from broadcast at ReadsSparkSink.java:133
17:38:09.758 WARN HtsjdkReadsRddStorage - Unrecognized write option: DISABLE
17:38:09.758 WARN HtsjdkReadsRddStorage - Unrecognized write option: DISABLE
17:38:09.759 INFO MemoryStore - Block broadcast_375 stored as values in memory (estimated size 163.2 KiB, free 1918.1 MiB)
17:38:09.759 INFO MemoryStore - Block broadcast_375_piece0 stored as bytes in memory (estimated size 9.6 KiB, free 1918.1 MiB)
17:38:09.759 INFO BlockManagerInfo - Added broadcast_375_piece0 in memory on localhost:37953 (size: 9.6 KiB, free: 1919.7 MiB)
17:38:09.760 INFO SparkContext - Created broadcast 375 from broadcast at BamSink.java:76
17:38:09.761 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:09.761 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:09.761 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:09.774 INFO SparkContext - Starting job: runJob at SparkHadoopWriter.scala:83
17:38:09.774 INFO DAGScheduler - Registering RDD 904 (mapToPair at SparkUtils.java:161) as input to shuffle 38
17:38:09.775 INFO DAGScheduler - Got job 141 (runJob at SparkHadoopWriter.scala:83) with 1 output partitions
17:38:09.775 INFO DAGScheduler - Final stage: ResultStage 191 (runJob at SparkHadoopWriter.scala:83)
17:38:09.775 INFO DAGScheduler - Parents of final stage: List(ShuffleMapStage 190)
17:38:09.775 INFO DAGScheduler - Missing parents: List(ShuffleMapStage 190)
17:38:09.775 INFO DAGScheduler - Submitting ShuffleMapStage 190 (MapPartitionsRDD[904] at mapToPair at SparkUtils.java:161), which has no missing parents
17:38:09.790 INFO MemoryStore - Block broadcast_376 stored as values in memory (estimated size 520.4 KiB, free 1917.6 MiB)
17:38:09.792 INFO MemoryStore - Block broadcast_376_piece0 stored as bytes in memory (estimated size 166.1 KiB, free 1917.4 MiB)
17:38:09.792 INFO BlockManagerInfo - Added broadcast_376_piece0 in memory on localhost:37953 (size: 166.1 KiB, free: 1919.6 MiB)
17:38:09.792 INFO SparkContext - Created broadcast 376 from broadcast at DAGScheduler.scala:1580
17:38:09.792 INFO DAGScheduler - Submitting 1 missing tasks from ShuffleMapStage 190 (MapPartitionsRDD[904] at mapToPair at SparkUtils.java:161) (first 15 tasks are for partitions Vector(0))
17:38:09.792 INFO TaskSchedulerImpl - Adding task set 190.0 with 1 tasks resource profile 0
17:38:09.793 INFO TaskSetManager - Starting task 0.0 in stage 190.0 (TID 246) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7861 bytes)
17:38:09.793 INFO Executor - Running task 0.0 in stage 190.0 (TID 246)
17:38:09.825 INFO NewHadoopRDD - Input split: file:/home/runner/work/gatk/gatk/src/test/resources/org/broadinstitute/hellbender/tools/BQSR/HiSeq.1mb.1RG.2k_lines.bam:0+222075
17:38:09.826 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:09.827 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:09.834 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:09.842 INFO Executor - Finished task 0.0 in stage 190.0 (TID 246). 1148 bytes result sent to driver
17:38:09.843 INFO TaskSetManager - Finished task 0.0 in stage 190.0 (TID 246) in 50 ms on localhost (executor driver) (1/1)
17:38:09.843 INFO TaskSchedulerImpl - Removed TaskSet 190.0, whose tasks have all completed, from pool
17:38:09.843 INFO DAGScheduler - ShuffleMapStage 190 (mapToPair at SparkUtils.java:161) finished in 0.068 s
17:38:09.843 INFO DAGScheduler - looking for newly runnable stages
17:38:09.843 INFO DAGScheduler - running: HashSet()
17:38:09.843 INFO DAGScheduler - waiting: HashSet(ResultStage 191)
17:38:09.843 INFO DAGScheduler - failed: HashSet()
17:38:09.843 INFO DAGScheduler - Submitting ResultStage 191 (MapPartitionsRDD[909] at mapToPair at BamSink.java:91), which has no missing parents
17:38:09.850 INFO MemoryStore - Block broadcast_377 stored as values in memory (estimated size 241.4 KiB, free 1917.2 MiB)
17:38:09.850 INFO MemoryStore - Block broadcast_377_piece0 stored as bytes in memory (estimated size 67.0 KiB, free 1917.1 MiB)
17:38:09.850 INFO BlockManagerInfo - Added broadcast_377_piece0 in memory on localhost:37953 (size: 67.0 KiB, free: 1919.5 MiB)
17:38:09.851 INFO SparkContext - Created broadcast 377 from broadcast at DAGScheduler.scala:1580
17:38:09.851 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 191 (MapPartitionsRDD[909] at mapToPair at BamSink.java:91) (first 15 tasks are for partitions Vector(0))
17:38:09.851 INFO TaskSchedulerImpl - Adding task set 191.0 with 1 tasks resource profile 0
17:38:09.851 INFO TaskSetManager - Starting task 0.0 in stage 191.0 (TID 247) (localhost, executor driver, partition 0, NODE_LOCAL, 7513 bytes)
17:38:09.852 INFO Executor - Running task 0.0 in stage 191.0 (TID 247)
17:38:09.856 INFO ShuffleBlockFetcherIterator - Getting 1 (343.8 KiB) non-empty blocks including 1 (343.8 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
17:38:09.857 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
17:38:09.868 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:09.868 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:09.868 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:09.868 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:09.868 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:09.868 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:09.875 WARN IntelDeflaterFactory - Intel Deflater not supported, using Java.util.zip.Deflater
17:38:09.884 INFO FileOutputCommitter - Saved output of task 'attempt_202510271738096243288860218355112_0909_r_000000_0' to file:/tmp/ReadsSparkSinkUnitTest1.someOtherPlace5700801713884919940/_temporary/0/task_202510271738096243288860218355112_0909_r_000000
17:38:09.884 INFO SparkHadoopMapRedUtil - attempt_202510271738096243288860218355112_0909_r_000000_0: Committed. Elapsed time: 0 ms.
17:38:09.885 INFO Executor - Finished task 0.0 in stage 191.0 (TID 247). 1858 bytes result sent to driver
17:38:09.885 INFO TaskSetManager - Finished task 0.0 in stage 191.0 (TID 247) in 34 ms on localhost (executor driver) (1/1)
17:38:09.885 INFO TaskSchedulerImpl - Removed TaskSet 191.0, whose tasks have all completed, from pool
17:38:09.885 INFO DAGScheduler - ResultStage 191 (runJob at SparkHadoopWriter.scala:83) finished in 0.041 s
17:38:09.885 INFO DAGScheduler - Job 141 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:09.885 INFO TaskSchedulerImpl - Killing all running tasks in stage 191: Stage finished
17:38:09.885 INFO DAGScheduler - Job 141 finished: runJob at SparkHadoopWriter.scala:83, took 0.111316 s
17:38:09.886 INFO SparkHadoopWriter - Start to commit write Job job_202510271738096243288860218355112_0909.
17:38:09.889 INFO SparkHadoopWriter - Write Job job_202510271738096243288860218355112_0909 committed. Elapsed time: 3 ms.
17:38:09.891 WARN IntelDeflaterFactory - Intel Deflater not supported, using Java.util.zip.Deflater
17:38:09.897 INFO HadoopFileSystemWrapper - Concatenating 3 parts to /tmp/ReadsSparkSinkUnitTest14320995241066527810.bam
17:38:09.900 INFO HadoopFileSystemWrapper - Concatenating to /tmp/ReadsSparkSinkUnitTest14320995241066527810.bam done
17:38:09.901 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:09.903 INFO MemoryStore - Block broadcast_378 stored as values in memory (estimated size 297.9 KiB, free 1916.8 MiB)
17:38:09.909 INFO BlockManagerInfo - Removed broadcast_361_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.6 MiB)
17:38:09.909 INFO BlockManagerInfo - Removed broadcast_371_piece0 on localhost:37953 in memory (size: 54.5 KiB, free: 1919.6 MiB)
17:38:09.910 INFO BlockManagerInfo - Removed broadcast_374_piece0 on localhost:37953 in memory (size: 9.6 KiB, free: 1919.6 MiB)
17:38:09.911 INFO BlockManagerInfo - Removed broadcast_368_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.7 MiB)
17:38:09.911 INFO BlockManagerInfo - Removed broadcast_373_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.7 MiB)
17:38:09.912 INFO BlockManagerInfo - Removed broadcast_375_piece0 on localhost:37953 in memory (size: 9.6 KiB, free: 1919.7 MiB)
17:38:09.912 INFO MemoryStore - Block broadcast_378_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1918.2 MiB)
17:38:09.912 INFO BlockManagerInfo - Added broadcast_378_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.7 MiB)
17:38:09.912 INFO SparkContext - Created broadcast 378 from newAPIHadoopFile at PathSplitSource.java:96
17:38:09.912 INFO BlockManagerInfo - Removed broadcast_367_piece0 on localhost:37953 in memory (size: 233.0 B, free: 1919.7 MiB)
17:38:09.913 INFO BlockManagerInfo - Removed broadcast_376_piece0 on localhost:37953 in memory (size: 166.1 KiB, free: 1919.8 MiB)
17:38:09.913 INFO BlockManagerInfo - Removed broadcast_377_piece0 on localhost:37953 in memory (size: 67.0 KiB, free: 1919.9 MiB)
17:38:09.931 INFO FileInputFormat - Total input files to process : 1
17:38:09.963 INFO SparkContext - Starting job: collect at ReadsSparkSinkUnitTest.java:182
17:38:09.964 INFO DAGScheduler - Got job 142 (collect at ReadsSparkSinkUnitTest.java:182) with 1 output partitions
17:38:09.964 INFO DAGScheduler - Final stage: ResultStage 192 (collect at ReadsSparkSinkUnitTest.java:182)
17:38:09.964 INFO DAGScheduler - Parents of final stage: List()
17:38:09.964 INFO DAGScheduler - Missing parents: List()
17:38:09.964 INFO DAGScheduler - Submitting ResultStage 192 (MapPartitionsRDD[916] at filter at ReadsSparkSource.java:96), which has no missing parents
17:38:09.979 INFO MemoryStore - Block broadcast_379 stored as values in memory (estimated size 426.2 KiB, free 1918.9 MiB)
17:38:09.980 INFO MemoryStore - Block broadcast_379_piece0 stored as bytes in memory (estimated size 153.7 KiB, free 1918.8 MiB)
17:38:09.980 INFO BlockManagerInfo - Added broadcast_379_piece0 in memory on localhost:37953 (size: 153.7 KiB, free: 1919.8 MiB)
17:38:09.981 INFO SparkContext - Created broadcast 379 from broadcast at DAGScheduler.scala:1580
17:38:09.981 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 192 (MapPartitionsRDD[916] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:38:09.981 INFO TaskSchedulerImpl - Adding task set 192.0 with 1 tasks resource profile 0
17:38:09.982 INFO TaskSetManager - Starting task 0.0 in stage 192.0 (TID 248) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7809 bytes)
17:38:09.982 INFO Executor - Running task 0.0 in stage 192.0 (TID 248)
17:38:10.013 INFO NewHadoopRDD - Input split: file:/tmp/ReadsSparkSinkUnitTest14320995241066527810.bam:0+238969
17:38:10.014 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:10.015 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:10.024 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:10.029 INFO Executor - Finished task 0.0 in stage 192.0 (TID 248). 651483 bytes result sent to driver
17:38:10.030 INFO TaskSetManager - Finished task 0.0 in stage 192.0 (TID 248) in 49 ms on localhost (executor driver) (1/1)
17:38:10.030 INFO TaskSchedulerImpl - Removed TaskSet 192.0, whose tasks have all completed, from pool
17:38:10.030 INFO DAGScheduler - ResultStage 192 (collect at ReadsSparkSinkUnitTest.java:182) finished in 0.066 s
17:38:10.030 INFO DAGScheduler - Job 142 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:10.030 INFO TaskSchedulerImpl - Killing all running tasks in stage 192: Stage finished
17:38:10.030 INFO DAGScheduler - Job 142 finished: collect at ReadsSparkSinkUnitTest.java:182, took 0.067141 s
17:38:10.040 INFO SparkContext - Starting job: count at ReadsSparkSinkUnitTest.java:185
17:38:10.041 INFO DAGScheduler - Got job 143 (count at ReadsSparkSinkUnitTest.java:185) with 1 output partitions
17:38:10.041 INFO DAGScheduler - Final stage: ResultStage 193 (count at ReadsSparkSinkUnitTest.java:185)
17:38:10.041 INFO DAGScheduler - Parents of final stage: List()
17:38:10.041 INFO DAGScheduler - Missing parents: List()
17:38:10.041 INFO DAGScheduler - Submitting ResultStage 193 (MapPartitionsRDD[897] at filter at ReadsSparkSource.java:96), which has no missing parents
17:38:10.056 INFO MemoryStore - Block broadcast_380 stored as values in memory (estimated size 426.1 KiB, free 1918.3 MiB)
17:38:10.057 INFO MemoryStore - Block broadcast_380_piece0 stored as bytes in memory (estimated size 153.6 KiB, free 1918.2 MiB)
17:38:10.057 INFO BlockManagerInfo - Added broadcast_380_piece0 in memory on localhost:37953 (size: 153.6 KiB, free: 1919.6 MiB)
17:38:10.058 INFO SparkContext - Created broadcast 380 from broadcast at DAGScheduler.scala:1580
17:38:10.058 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 193 (MapPartitionsRDD[897] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:38:10.058 INFO TaskSchedulerImpl - Adding task set 193.0 with 1 tasks resource profile 0
17:38:10.058 INFO TaskSetManager - Starting task 0.0 in stage 193.0 (TID 249) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7872 bytes)
17:38:10.059 INFO Executor - Running task 0.0 in stage 193.0 (TID 249)
17:38:10.090 INFO NewHadoopRDD - Input split: file:/home/runner/work/gatk/gatk/src/test/resources/org/broadinstitute/hellbender/tools/BQSR/HiSeq.1mb.1RG.2k_lines.bam:0+222075
17:38:10.090 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:10.091 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:10.098 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:10.102 INFO Executor - Finished task 0.0 in stage 193.0 (TID 249). 989 bytes result sent to driver
17:38:10.102 INFO TaskSetManager - Finished task 0.0 in stage 193.0 (TID 249) in 44 ms on localhost (executor driver) (1/1)
17:38:10.102 INFO TaskSchedulerImpl - Removed TaskSet 193.0, whose tasks have all completed, from pool
17:38:10.102 INFO DAGScheduler - ResultStage 193 (count at ReadsSparkSinkUnitTest.java:185) finished in 0.061 s
17:38:10.102 INFO DAGScheduler - Job 143 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:10.102 INFO TaskSchedulerImpl - Killing all running tasks in stage 193: Stage finished
17:38:10.102 INFO DAGScheduler - Job 143 finished: count at ReadsSparkSinkUnitTest.java:185, took 0.062109 s
17:38:10.106 INFO SparkContext - Starting job: count at ReadsSparkSinkUnitTest.java:185
17:38:10.106 INFO DAGScheduler - Got job 144 (count at ReadsSparkSinkUnitTest.java:185) with 1 output partitions
17:38:10.106 INFO DAGScheduler - Final stage: ResultStage 194 (count at ReadsSparkSinkUnitTest.java:185)
17:38:10.106 INFO DAGScheduler - Parents of final stage: List()
17:38:10.106 INFO DAGScheduler - Missing parents: List()
17:38:10.106 INFO DAGScheduler - Submitting ResultStage 194 (MapPartitionsRDD[916] at filter at ReadsSparkSource.java:96), which has no missing parents
17:38:10.121 INFO MemoryStore - Block broadcast_381 stored as values in memory (estimated size 426.1 KiB, free 1917.8 MiB)
17:38:10.122 INFO MemoryStore - Block broadcast_381_piece0 stored as bytes in memory (estimated size 153.6 KiB, free 1917.6 MiB)
17:38:10.122 INFO BlockManagerInfo - Added broadcast_381_piece0 in memory on localhost:37953 (size: 153.6 KiB, free: 1919.5 MiB)
17:38:10.123 INFO SparkContext - Created broadcast 381 from broadcast at DAGScheduler.scala:1580
17:38:10.123 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 194 (MapPartitionsRDD[916] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:38:10.123 INFO TaskSchedulerImpl - Adding task set 194.0 with 1 tasks resource profile 0
17:38:10.123 INFO TaskSetManager - Starting task 0.0 in stage 194.0 (TID 250) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7809 bytes)
17:38:10.124 INFO Executor - Running task 0.0 in stage 194.0 (TID 250)
17:38:10.154 INFO NewHadoopRDD - Input split: file:/tmp/ReadsSparkSinkUnitTest14320995241066527810.bam:0+238969
17:38:10.154 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:10.155 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:10.164 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:10.168 INFO Executor - Finished task 0.0 in stage 194.0 (TID 250). 989 bytes result sent to driver
17:38:10.168 INFO TaskSetManager - Finished task 0.0 in stage 194.0 (TID 250) in 45 ms on localhost (executor driver) (1/1)
17:38:10.168 INFO TaskSchedulerImpl - Removed TaskSet 194.0, whose tasks have all completed, from pool
17:38:10.168 INFO DAGScheduler - ResultStage 194 (count at ReadsSparkSinkUnitTest.java:185) finished in 0.062 s
17:38:10.168 INFO DAGScheduler - Job 144 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:10.168 INFO TaskSchedulerImpl - Killing all running tasks in stage 194: Stage finished
17:38:10.168 INFO DAGScheduler - Job 144 finished: count at ReadsSparkSinkUnitTest.java:185, took 0.062863 s
17:38:10.173 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:10.175 INFO MemoryStore - Block broadcast_382 stored as values in memory (estimated size 298.0 KiB, free 1917.3 MiB)
17:38:10.180 INFO MemoryStore - Block broadcast_382_piece0 stored as bytes in memory (estimated size 50.3 KiB, free 1917.3 MiB)
17:38:10.181 INFO BlockManagerInfo - Added broadcast_382_piece0 in memory on localhost:37953 (size: 50.3 KiB, free: 1919.4 MiB)
17:38:10.181 INFO SparkContext - Created broadcast 382 from newAPIHadoopFile at PathSplitSource.java:96
17:38:10.200 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:10.202 INFO MemoryStore - Block broadcast_383 stored as values in memory (estimated size 298.0 KiB, free 1917.0 MiB)
17:38:10.207 INFO MemoryStore - Block broadcast_383_piece0 stored as bytes in memory (estimated size 50.3 KiB, free 1916.9 MiB)
17:38:10.207 INFO BlockManagerInfo - Added broadcast_383_piece0 in memory on localhost:37953 (size: 50.3 KiB, free: 1919.4 MiB)
17:38:10.207 INFO SparkContext - Created broadcast 383 from newAPIHadoopFile at PathSplitSource.java:96
17:38:10.226 INFO FileInputFormat - Total input files to process : 1
17:38:10.228 INFO MemoryStore - Block broadcast_384 stored as values in memory (estimated size 160.7 KiB, free 1916.8 MiB)
17:38:10.232 INFO MemoryStore - Block broadcast_384_piece0 stored as bytes in memory (estimated size 9.6 KiB, free 1917.3 MiB)
17:38:10.232 INFO BlockManagerInfo - Removed broadcast_380_piece0 on localhost:37953 in memory (size: 153.6 KiB, free: 1919.5 MiB)
17:38:10.233 INFO BlockManagerInfo - Added broadcast_384_piece0 in memory on localhost:37953 (size: 9.6 KiB, free: 1919.5 MiB)
17:38:10.233 INFO SparkContext - Created broadcast 384 from broadcast at ReadsSparkSink.java:133
17:38:10.233 INFO BlockManagerInfo - Removed broadcast_383_piece0 on localhost:37953 in memory (size: 50.3 KiB, free: 1919.5 MiB)
17:38:10.233 INFO BlockManagerInfo - Removed broadcast_372_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.6 MiB)
17:38:10.234 INFO BlockManagerInfo - Removed broadcast_378_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.6 MiB)
17:38:10.234 INFO MemoryStore - Block broadcast_385 stored as values in memory (estimated size 163.2 KiB, free 1918.2 MiB)
17:38:10.234 INFO BlockManagerInfo - Removed broadcast_379_piece0 on localhost:37953 in memory (size: 153.7 KiB, free: 1919.8 MiB)
17:38:10.235 INFO MemoryStore - Block broadcast_385_piece0 stored as bytes in memory (estimated size 9.6 KiB, free 1918.9 MiB)
17:38:10.235 INFO BlockManagerInfo - Removed broadcast_381_piece0 on localhost:37953 in memory (size: 153.6 KiB, free: 1919.9 MiB)
17:38:10.235 INFO BlockManagerInfo - Added broadcast_385_piece0 in memory on localhost:37953 (size: 9.6 KiB, free: 1919.9 MiB)
17:38:10.235 INFO SparkContext - Created broadcast 385 from broadcast at BamSink.java:76
17:38:10.237 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:10.237 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:10.237 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:10.250 INFO SparkContext - Starting job: runJob at SparkHadoopWriter.scala:83
17:38:10.250 INFO DAGScheduler - Registering RDD 930 (mapToPair at SparkUtils.java:161) as input to shuffle 39
17:38:10.250 INFO DAGScheduler - Got job 145 (runJob at SparkHadoopWriter.scala:83) with 1 output partitions
17:38:10.250 INFO DAGScheduler - Final stage: ResultStage 196 (runJob at SparkHadoopWriter.scala:83)
17:38:10.250 INFO DAGScheduler - Parents of final stage: List(ShuffleMapStage 195)
17:38:10.250 INFO DAGScheduler - Missing parents: List(ShuffleMapStage 195)
17:38:10.250 INFO DAGScheduler - Submitting ShuffleMapStage 195 (MapPartitionsRDD[930] at mapToPair at SparkUtils.java:161), which has no missing parents
17:38:10.266 INFO MemoryStore - Block broadcast_386 stored as values in memory (estimated size 520.4 KiB, free 1918.8 MiB)
17:38:10.267 INFO MemoryStore - Block broadcast_386_piece0 stored as bytes in memory (estimated size 166.1 KiB, free 1918.7 MiB)
17:38:10.267 INFO BlockManagerInfo - Added broadcast_386_piece0 in memory on localhost:37953 (size: 166.1 KiB, free: 1919.8 MiB)
17:38:10.268 INFO SparkContext - Created broadcast 386 from broadcast at DAGScheduler.scala:1580
17:38:10.268 INFO DAGScheduler - Submitting 1 missing tasks from ShuffleMapStage 195 (MapPartitionsRDD[930] at mapToPair at SparkUtils.java:161) (first 15 tasks are for partitions Vector(0))
17:38:10.268 INFO TaskSchedulerImpl - Adding task set 195.0 with 1 tasks resource profile 0
17:38:10.268 INFO TaskSetManager - Starting task 0.0 in stage 195.0 (TID 251) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7901 bytes)
17:38:10.269 INFO Executor - Running task 0.0 in stage 195.0 (TID 251)
17:38:10.300 INFO NewHadoopRDD - Input split: file:/home/runner/work/gatk/gatk/src/test/resources/org/broadinstitute/hellbender/tools/BQSR/expected.HiSeq.1mb.1RG.2k_lines.alternate.recalibrated.DIQ.bam:0+216896
17:38:10.301 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:10.302 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:10.311 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:10.320 INFO Executor - Finished task 0.0 in stage 195.0 (TID 251). 1148 bytes result sent to driver
17:38:10.320 INFO TaskSetManager - Finished task 0.0 in stage 195.0 (TID 251) in 52 ms on localhost (executor driver) (1/1)
17:38:10.320 INFO TaskSchedulerImpl - Removed TaskSet 195.0, whose tasks have all completed, from pool
17:38:10.320 INFO DAGScheduler - ShuffleMapStage 195 (mapToPair at SparkUtils.java:161) finished in 0.069 s
17:38:10.320 INFO DAGScheduler - looking for newly runnable stages
17:38:10.320 INFO DAGScheduler - running: HashSet()
17:38:10.320 INFO DAGScheduler - waiting: HashSet(ResultStage 196)
17:38:10.320 INFO DAGScheduler - failed: HashSet()
17:38:10.320 INFO DAGScheduler - Submitting ResultStage 196 (MapPartitionsRDD[935] at mapToPair at BamSink.java:91), which has no missing parents
17:38:10.327 INFO MemoryStore - Block broadcast_387 stored as values in memory (estimated size 241.4 KiB, free 1918.4 MiB)
17:38:10.327 INFO MemoryStore - Block broadcast_387_piece0 stored as bytes in memory (estimated size 67.0 KiB, free 1918.4 MiB)
17:38:10.327 INFO BlockManagerInfo - Added broadcast_387_piece0 in memory on localhost:37953 (size: 67.0 KiB, free: 1919.7 MiB)
17:38:10.327 INFO SparkContext - Created broadcast 387 from broadcast at DAGScheduler.scala:1580
17:38:10.328 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 196 (MapPartitionsRDD[935] at mapToPair at BamSink.java:91) (first 15 tasks are for partitions Vector(0))
17:38:10.328 INFO TaskSchedulerImpl - Adding task set 196.0 with 1 tasks resource profile 0
17:38:10.328 INFO TaskSetManager - Starting task 0.0 in stage 196.0 (TID 252) (localhost, executor driver, partition 0, NODE_LOCAL, 7513 bytes)
17:38:10.328 INFO Executor - Running task 0.0 in stage 196.0 (TID 252)
17:38:10.333 INFO ShuffleBlockFetcherIterator - Getting 1 (343.8 KiB) non-empty blocks including 1 (343.8 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
17:38:10.333 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
17:38:10.344 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:10.344 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:10.344 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:10.345 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:10.345 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:10.345 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:10.352 WARN IntelDeflaterFactory - Intel Deflater not supported, using Java.util.zip.Deflater
17:38:10.368 INFO FileOutputCommitter - Saved output of task 'attempt_202510271738102949861632817389055_0935_r_000000_0' to file:/tmp/ReadsSparkSinkUnitTest2.someOtherPlace16189953178008125193/_temporary/0/task_202510271738102949861632817389055_0935_r_000000
17:38:10.368 INFO SparkHadoopMapRedUtil - attempt_202510271738102949861632817389055_0935_r_000000_0: Committed. Elapsed time: 0 ms.
17:38:10.368 INFO Executor - Finished task 0.0 in stage 196.0 (TID 252). 1858 bytes result sent to driver
17:38:10.368 INFO TaskSetManager - Finished task 0.0 in stage 196.0 (TID 252) in 40 ms on localhost (executor driver) (1/1)
17:38:10.369 INFO TaskSchedulerImpl - Removed TaskSet 196.0, whose tasks have all completed, from pool
17:38:10.369 INFO DAGScheduler - ResultStage 196 (runJob at SparkHadoopWriter.scala:83) finished in 0.048 s
17:38:10.369 INFO DAGScheduler - Job 145 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:10.369 INFO TaskSchedulerImpl - Killing all running tasks in stage 196: Stage finished
17:38:10.369 INFO DAGScheduler - Job 145 finished: runJob at SparkHadoopWriter.scala:83, took 0.119431 s
17:38:10.370 INFO SparkHadoopWriter - Start to commit write Job job_202510271738102949861632817389055_0935.
17:38:10.373 INFO SparkHadoopWriter - Write Job job_202510271738102949861632817389055_0935 committed. Elapsed time: 3 ms.
17:38:10.375 WARN IntelDeflaterFactory - Intel Deflater not supported, using Java.util.zip.Deflater
17:38:10.381 INFO HadoopFileSystemWrapper - Concatenating 3 parts to /tmp/ReadsSparkSinkUnitTest210938238991699765791.bam
17:38:10.384 INFO HadoopFileSystemWrapper - Concatenating to /tmp/ReadsSparkSinkUnitTest210938238991699765791.bam done
17:38:10.384 INFO IndexFileMerger - Merging .sbi files in temp directory /tmp/ReadsSparkSinkUnitTest2.someOtherPlace16189953178008125193 to /tmp/ReadsSparkSinkUnitTest210938238991699765791.bam.sbi
17:38:10.389 INFO IndexFileMerger - Done merging .sbi files
17:38:10.389 INFO IndexFileMerger - Merging .bai files in temp directory /tmp/ReadsSparkSinkUnitTest2.someOtherPlace16189953178008125193 to /tmp/ReadsSparkSinkUnitTest210938238991699765791.bam.bai
17:38:10.392 INFO IndexFileMerger - Done merging .bai files
17:38:10.393 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:10.394 INFO MemoryStore - Block broadcast_388 stored as values in memory (estimated size 320.0 B, free 1918.4 MiB)
17:38:10.395 INFO MemoryStore - Block broadcast_388_piece0 stored as bytes in memory (estimated size 233.0 B, free 1918.4 MiB)
17:38:10.395 INFO BlockManagerInfo - Added broadcast_388_piece0 in memory on localhost:37953 (size: 233.0 B, free: 1919.7 MiB)
17:38:10.395 INFO SparkContext - Created broadcast 388 from broadcast at BamSource.java:104
17:38:10.396 INFO MemoryStore - Block broadcast_389 stored as values in memory (estimated size 297.9 KiB, free 1918.1 MiB)
17:38:10.402 INFO MemoryStore - Block broadcast_389_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1918.0 MiB)
17:38:10.402 INFO BlockManagerInfo - Added broadcast_389_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.7 MiB)
17:38:10.402 INFO SparkContext - Created broadcast 389 from newAPIHadoopFile at PathSplitSource.java:96
17:38:10.410 INFO FileInputFormat - Total input files to process : 1
17:38:10.423 INFO SparkContext - Starting job: collect at ReadsSparkSinkUnitTest.java:182
17:38:10.423 INFO DAGScheduler - Got job 146 (collect at ReadsSparkSinkUnitTest.java:182) with 1 output partitions
17:38:10.423 INFO DAGScheduler - Final stage: ResultStage 197 (collect at ReadsSparkSinkUnitTest.java:182)
17:38:10.424 INFO DAGScheduler - Parents of final stage: List()
17:38:10.424 INFO DAGScheduler - Missing parents: List()
17:38:10.424 INFO DAGScheduler - Submitting ResultStage 197 (MapPartitionsRDD[941] at filter at ReadsSparkSource.java:96), which has no missing parents
17:38:10.429 INFO MemoryStore - Block broadcast_390 stored as values in memory (estimated size 148.2 KiB, free 1917.9 MiB)
17:38:10.432 INFO MemoryStore - Block broadcast_390_piece0 stored as bytes in memory (estimated size 54.5 KiB, free 1917.8 MiB)
17:38:10.433 INFO BlockManagerInfo - Added broadcast_390_piece0 in memory on localhost:37953 (size: 54.5 KiB, free: 1919.6 MiB)
17:38:10.433 INFO SparkContext - Created broadcast 390 from broadcast at DAGScheduler.scala:1580
17:38:10.433 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 197 (MapPartitionsRDD[941] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:38:10.433 INFO TaskSchedulerImpl - Adding task set 197.0 with 1 tasks resource profile 0
17:38:10.433 INFO BlockManagerInfo - Removed broadcast_385_piece0 on localhost:37953 in memory (size: 9.6 KiB, free: 1919.6 MiB)
17:38:10.434 INFO BlockManagerInfo - Removed broadcast_384_piece0 on localhost:37953 in memory (size: 9.6 KiB, free: 1919.6 MiB)
17:38:10.434 INFO TaskSetManager - Starting task 0.0 in stage 197.0 (TID 253) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7810 bytes)
17:38:10.435 INFO Executor - Running task 0.0 in stage 197.0 (TID 253)
17:38:10.438 INFO BlockManagerInfo - Removed broadcast_387_piece0 on localhost:37953 in memory (size: 67.0 KiB, free: 1919.7 MiB)
17:38:10.438 INFO BlockManagerInfo - Removed broadcast_386_piece0 on localhost:37953 in memory (size: 166.1 KiB, free: 1919.8 MiB)
17:38:10.448 INFO NewHadoopRDD - Input split: file:/tmp/ReadsSparkSinkUnitTest210938238991699765791.bam:0+236321
17:38:10.449 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:10.453 INFO Executor - Finished task 0.0 in stage 197.0 (TID 253). 650141 bytes result sent to driver
17:38:10.454 INFO TaskSetManager - Finished task 0.0 in stage 197.0 (TID 253) in 20 ms on localhost (executor driver) (1/1)
17:38:10.454 INFO TaskSchedulerImpl - Removed TaskSet 197.0, whose tasks have all completed, from pool
17:38:10.454 INFO DAGScheduler - ResultStage 197 (collect at ReadsSparkSinkUnitTest.java:182) finished in 0.030 s
17:38:10.455 INFO DAGScheduler - Job 146 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:10.455 INFO TaskSchedulerImpl - Killing all running tasks in stage 197: Stage finished
17:38:10.455 INFO DAGScheduler - Job 146 finished: collect at ReadsSparkSinkUnitTest.java:182, took 0.031548 s
17:38:10.464 INFO SparkContext - Starting job: count at ReadsSparkSinkUnitTest.java:185
17:38:10.465 INFO DAGScheduler - Got job 147 (count at ReadsSparkSinkUnitTest.java:185) with 1 output partitions
17:38:10.465 INFO DAGScheduler - Final stage: ResultStage 198 (count at ReadsSparkSinkUnitTest.java:185)
17:38:10.465 INFO DAGScheduler - Parents of final stage: List()
17:38:10.465 INFO DAGScheduler - Missing parents: List()
17:38:10.465 INFO DAGScheduler - Submitting ResultStage 198 (MapPartitionsRDD[923] at filter at ReadsSparkSource.java:96), which has no missing parents
17:38:10.480 INFO MemoryStore - Block broadcast_391 stored as values in memory (estimated size 426.1 KiB, free 1918.7 MiB)
17:38:10.481 INFO MemoryStore - Block broadcast_391_piece0 stored as bytes in memory (estimated size 153.6 KiB, free 1918.6 MiB)
17:38:10.481 INFO BlockManagerInfo - Added broadcast_391_piece0 in memory on localhost:37953 (size: 153.6 KiB, free: 1919.7 MiB)
17:38:10.482 INFO SparkContext - Created broadcast 391 from broadcast at DAGScheduler.scala:1580
17:38:10.482 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 198 (MapPartitionsRDD[923] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:38:10.482 INFO TaskSchedulerImpl - Adding task set 198.0 with 1 tasks resource profile 0
17:38:10.482 INFO TaskSetManager - Starting task 0.0 in stage 198.0 (TID 254) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7912 bytes)
17:38:10.483 INFO Executor - Running task 0.0 in stage 198.0 (TID 254)
17:38:10.514 INFO NewHadoopRDD - Input split: file:/home/runner/work/gatk/gatk/src/test/resources/org/broadinstitute/hellbender/tools/BQSR/expected.HiSeq.1mb.1RG.2k_lines.alternate.recalibrated.DIQ.bam:0+216896
17:38:10.515 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:10.516 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:10.525 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:10.528 INFO Executor - Finished task 0.0 in stage 198.0 (TID 254). 989 bytes result sent to driver
17:38:10.529 INFO TaskSetManager - Finished task 0.0 in stage 198.0 (TID 254) in 47 ms on localhost (executor driver) (1/1)
17:38:10.529 INFO TaskSchedulerImpl - Removed TaskSet 198.0, whose tasks have all completed, from pool
17:38:10.529 INFO DAGScheduler - ResultStage 198 (count at ReadsSparkSinkUnitTest.java:185) finished in 0.064 s
17:38:10.529 INFO DAGScheduler - Job 147 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:10.529 INFO TaskSchedulerImpl - Killing all running tasks in stage 198: Stage finished
17:38:10.529 INFO DAGScheduler - Job 147 finished: count at ReadsSparkSinkUnitTest.java:185, took 0.064535 s
17:38:10.532 INFO SparkContext - Starting job: count at ReadsSparkSinkUnitTest.java:185
17:38:10.532 INFO DAGScheduler - Got job 148 (count at ReadsSparkSinkUnitTest.java:185) with 1 output partitions
17:38:10.532 INFO DAGScheduler - Final stage: ResultStage 199 (count at ReadsSparkSinkUnitTest.java:185)
17:38:10.532 INFO DAGScheduler - Parents of final stage: List()
17:38:10.532 INFO DAGScheduler - Missing parents: List()
17:38:10.532 INFO DAGScheduler - Submitting ResultStage 199 (MapPartitionsRDD[941] at filter at ReadsSparkSource.java:96), which has no missing parents
17:38:10.538 INFO MemoryStore - Block broadcast_392 stored as values in memory (estimated size 148.1 KiB, free 1918.4 MiB)
17:38:10.539 INFO MemoryStore - Block broadcast_392_piece0 stored as bytes in memory (estimated size 54.5 KiB, free 1918.4 MiB)
17:38:10.539 INFO BlockManagerInfo - Added broadcast_392_piece0 in memory on localhost:37953 (size: 54.5 KiB, free: 1919.6 MiB)
17:38:10.539 INFO SparkContext - Created broadcast 392 from broadcast at DAGScheduler.scala:1580
17:38:10.539 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 199 (MapPartitionsRDD[941] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:38:10.539 INFO TaskSchedulerImpl - Adding task set 199.0 with 1 tasks resource profile 0
17:38:10.540 INFO TaskSetManager - Starting task 0.0 in stage 199.0 (TID 255) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7810 bytes)
17:38:10.540 INFO Executor - Running task 0.0 in stage 199.0 (TID 255)
17:38:10.551 INFO NewHadoopRDD - Input split: file:/tmp/ReadsSparkSinkUnitTest210938238991699765791.bam:0+236321
17:38:10.551 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:10.555 INFO Executor - Finished task 0.0 in stage 199.0 (TID 255). 989 bytes result sent to driver
17:38:10.555 INFO TaskSetManager - Finished task 0.0 in stage 199.0 (TID 255) in 16 ms on localhost (executor driver) (1/1)
17:38:10.555 INFO TaskSchedulerImpl - Removed TaskSet 199.0, whose tasks have all completed, from pool
17:38:10.556 INFO DAGScheduler - ResultStage 199 (count at ReadsSparkSinkUnitTest.java:185) finished in 0.022 s
17:38:10.556 INFO DAGScheduler - Job 148 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:10.556 INFO TaskSchedulerImpl - Killing all running tasks in stage 199: Stage finished
17:38:10.556 INFO DAGScheduler - Job 148 finished: count at ReadsSparkSinkUnitTest.java:185, took 0.023650 s
17:38:10.560 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:10.562 INFO MemoryStore - Block broadcast_393 stored as values in memory (estimated size 298.0 KiB, free 1918.1 MiB)
17:38:10.567 INFO MemoryStore - Block broadcast_393_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1918.0 MiB)
17:38:10.567 INFO BlockManagerInfo - Added broadcast_393_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.6 MiB)
17:38:10.567 INFO SparkContext - Created broadcast 393 from newAPIHadoopFile at PathSplitSource.java:96
17:38:10.586 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:10.588 INFO MemoryStore - Block broadcast_394 stored as values in memory (estimated size 298.0 KiB, free 1917.7 MiB)
17:38:10.593 INFO MemoryStore - Block broadcast_394_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1917.7 MiB)
17:38:10.593 INFO BlockManagerInfo - Added broadcast_394_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.5 MiB)
17:38:10.594 INFO SparkContext - Created broadcast 394 from newAPIHadoopFile at PathSplitSource.java:96
17:38:10.612 INFO FileInputFormat - Total input files to process : 1
17:38:10.614 INFO MemoryStore - Block broadcast_395 stored as values in memory (estimated size 19.6 KiB, free 1917.7 MiB)
17:38:10.614 INFO MemoryStore - Block broadcast_395_piece0 stored as bytes in memory (estimated size 1890.0 B, free 1917.7 MiB)
17:38:10.614 INFO BlockManagerInfo - Added broadcast_395_piece0 in memory on localhost:37953 (size: 1890.0 B, free: 1919.5 MiB)
17:38:10.614 INFO SparkContext - Created broadcast 395 from broadcast at ReadsSparkSink.java:133
17:38:10.615 INFO MemoryStore - Block broadcast_396 stored as values in memory (estimated size 20.0 KiB, free 1917.6 MiB)
17:38:10.618 INFO MemoryStore - Block broadcast_396_piece0 stored as bytes in memory (estimated size 1890.0 B, free 1917.6 MiB)
17:38:10.618 INFO BlockManagerInfo - Added broadcast_396_piece0 in memory on localhost:37953 (size: 1890.0 B, free: 1919.5 MiB)
17:38:10.619 INFO BlockManagerInfo - Removed broadcast_390_piece0 on localhost:37953 in memory (size: 54.5 KiB, free: 1919.6 MiB)
17:38:10.619 INFO SparkContext - Created broadcast 396 from broadcast at BamSink.java:76
17:38:10.620 INFO BlockManagerInfo - Removed broadcast_389_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.6 MiB)
17:38:10.620 INFO BlockManagerInfo - Removed broadcast_391_piece0 on localhost:37953 in memory (size: 153.6 KiB, free: 1919.8 MiB)
17:38:10.620 INFO BlockManagerInfo - Removed broadcast_392_piece0 on localhost:37953 in memory (size: 54.5 KiB, free: 1919.8 MiB)
17:38:10.621 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:10.621 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:10.621 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:10.621 INFO BlockManagerInfo - Removed broadcast_394_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.9 MiB)
17:38:10.621 INFO BlockManagerInfo - Removed broadcast_382_piece0 on localhost:37953 in memory (size: 50.3 KiB, free: 1919.9 MiB)
17:38:10.622 INFO BlockManagerInfo - Removed broadcast_388_piece0 on localhost:37953 in memory (size: 233.0 B, free: 1919.9 MiB)
17:38:10.633 INFO SparkContext - Starting job: runJob at SparkHadoopWriter.scala:83
17:38:10.634 INFO DAGScheduler - Registering RDD 955 (mapToPair at SparkUtils.java:161) as input to shuffle 40
17:38:10.634 INFO DAGScheduler - Got job 149 (runJob at SparkHadoopWriter.scala:83) with 1 output partitions
17:38:10.634 INFO DAGScheduler - Final stage: ResultStage 201 (runJob at SparkHadoopWriter.scala:83)
17:38:10.634 INFO DAGScheduler - Parents of final stage: List(ShuffleMapStage 200)
17:38:10.634 INFO DAGScheduler - Missing parents: List(ShuffleMapStage 200)
17:38:10.634 INFO DAGScheduler - Submitting ShuffleMapStage 200 (MapPartitionsRDD[955] at mapToPair at SparkUtils.java:161), which has no missing parents
17:38:10.649 INFO MemoryStore - Block broadcast_397 stored as values in memory (estimated size 434.3 KiB, free 1919.2 MiB)
17:38:10.651 INFO MemoryStore - Block broadcast_397_piece0 stored as bytes in memory (estimated size 157.6 KiB, free 1919.0 MiB)
17:38:10.651 INFO BlockManagerInfo - Added broadcast_397_piece0 in memory on localhost:37953 (size: 157.6 KiB, free: 1919.8 MiB)
17:38:10.651 INFO SparkContext - Created broadcast 397 from broadcast at DAGScheduler.scala:1580
17:38:10.651 INFO DAGScheduler - Submitting 1 missing tasks from ShuffleMapStage 200 (MapPartitionsRDD[955] at mapToPair at SparkUtils.java:161) (first 15 tasks are for partitions Vector(0))
17:38:10.651 INFO TaskSchedulerImpl - Adding task set 200.0 with 1 tasks resource profile 0
17:38:10.652 INFO TaskSetManager - Starting task 0.0 in stage 200.0 (TID 256) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7882 bytes)
17:38:10.652 INFO Executor - Running task 0.0 in stage 200.0 (TID 256)
17:38:10.683 INFO NewHadoopRDD - Input split: file:/home/runner/work/gatk/gatk/src/test/resources/org/broadinstitute/hellbender/tools/BQSR/CEUTrio.HiSeq.WGS.b37.ch20.1m-1m1k.NA12878.bam:0+211123
17:38:10.684 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:10.685 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:10.690 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:10.698 INFO Executor - Finished task 0.0 in stage 200.0 (TID 256). 1148 bytes result sent to driver
17:38:10.698 INFO TaskSetManager - Finished task 0.0 in stage 200.0 (TID 256) in 46 ms on localhost (executor driver) (1/1)
17:38:10.698 INFO TaskSchedulerImpl - Removed TaskSet 200.0, whose tasks have all completed, from pool
17:38:10.699 INFO DAGScheduler - ShuffleMapStage 200 (mapToPair at SparkUtils.java:161) finished in 0.065 s
17:38:10.699 INFO DAGScheduler - looking for newly runnable stages
17:38:10.699 INFO DAGScheduler - running: HashSet()
17:38:10.699 INFO DAGScheduler - waiting: HashSet(ResultStage 201)
17:38:10.699 INFO DAGScheduler - failed: HashSet()
17:38:10.699 INFO DAGScheduler - Submitting ResultStage 201 (MapPartitionsRDD[960] at mapToPair at BamSink.java:91), which has no missing parents
17:38:10.705 INFO MemoryStore - Block broadcast_398 stored as values in memory (estimated size 155.3 KiB, free 1918.9 MiB)
17:38:10.705 INFO MemoryStore - Block broadcast_398_piece0 stored as bytes in memory (estimated size 58.5 KiB, free 1918.8 MiB)
17:38:10.705 INFO BlockManagerInfo - Added broadcast_398_piece0 in memory on localhost:37953 (size: 58.5 KiB, free: 1919.7 MiB)
17:38:10.706 INFO SparkContext - Created broadcast 398 from broadcast at DAGScheduler.scala:1580
17:38:10.706 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 201 (MapPartitionsRDD[960] at mapToPair at BamSink.java:91) (first 15 tasks are for partitions Vector(0))
17:38:10.706 INFO TaskSchedulerImpl - Adding task set 201.0 with 1 tasks resource profile 0
17:38:10.706 INFO TaskSetManager - Starting task 0.0 in stage 201.0 (TID 257) (localhost, executor driver, partition 0, NODE_LOCAL, 7513 bytes)
17:38:10.706 INFO Executor - Running task 0.0 in stage 201.0 (TID 257)
17:38:10.711 INFO ShuffleBlockFetcherIterator - Getting 1 (312.6 KiB) non-empty blocks including 1 (312.6 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
17:38:10.711 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
17:38:10.722 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:10.722 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:10.722 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:10.722 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:10.722 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:10.722 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:10.730 WARN IntelDeflaterFactory - Intel Deflater not supported, using Java.util.zip.Deflater
17:38:10.745 INFO FileOutputCommitter - Saved output of task 'attempt_202510271738107040431760069792027_0960_r_000000_0' to file:/tmp/ReadsSparkSinkUnitTest3.someOtherPlace12298833529200217212/_temporary/0/task_202510271738107040431760069792027_0960_r_000000
17:38:10.745 INFO SparkHadoopMapRedUtil - attempt_202510271738107040431760069792027_0960_r_000000_0: Committed. Elapsed time: 0 ms.
17:38:10.745 INFO Executor - Finished task 0.0 in stage 201.0 (TID 257). 1858 bytes result sent to driver
17:38:10.746 INFO TaskSetManager - Finished task 0.0 in stage 201.0 (TID 257) in 40 ms on localhost (executor driver) (1/1)
17:38:10.746 INFO TaskSchedulerImpl - Removed TaskSet 201.0, whose tasks have all completed, from pool
17:38:10.746 INFO DAGScheduler - ResultStage 201 (runJob at SparkHadoopWriter.scala:83) finished in 0.047 s
17:38:10.746 INFO DAGScheduler - Job 149 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:10.746 INFO TaskSchedulerImpl - Killing all running tasks in stage 201: Stage finished
17:38:10.746 INFO DAGScheduler - Job 149 finished: runJob at SparkHadoopWriter.scala:83, took 0.112989 s
17:38:10.747 INFO SparkHadoopWriter - Start to commit write Job job_202510271738107040431760069792027_0960.
17:38:10.750 INFO SparkHadoopWriter - Write Job job_202510271738107040431760069792027_0960 committed. Elapsed time: 3 ms.
17:38:10.752 WARN IntelDeflaterFactory - Intel Deflater not supported, using Java.util.zip.Deflater
17:38:10.757 INFO HadoopFileSystemWrapper - Concatenating 3 parts to /tmp/ReadsSparkSinkUnitTest316721804858257364706.bam
17:38:10.760 INFO HadoopFileSystemWrapper - Concatenating to /tmp/ReadsSparkSinkUnitTest316721804858257364706.bam done
17:38:10.760 INFO IndexFileMerger - Merging .sbi files in temp directory /tmp/ReadsSparkSinkUnitTest3.someOtherPlace12298833529200217212 to /tmp/ReadsSparkSinkUnitTest316721804858257364706.bam.sbi
17:38:10.765 INFO IndexFileMerger - Done merging .sbi files
17:38:10.765 INFO IndexFileMerger - Merging .bai files in temp directory /tmp/ReadsSparkSinkUnitTest3.someOtherPlace12298833529200217212 to /tmp/ReadsSparkSinkUnitTest316721804858257364706.bam.bai
17:38:10.768 INFO IndexFileMerger - Done merging .bai files
17:38:10.769 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:10.770 INFO MemoryStore - Block broadcast_399 stored as values in memory (estimated size 312.0 B, free 1918.8 MiB)
17:38:10.770 INFO MemoryStore - Block broadcast_399_piece0 stored as bytes in memory (estimated size 231.0 B, free 1918.8 MiB)
17:38:10.770 INFO BlockManagerInfo - Added broadcast_399_piece0 in memory on localhost:37953 (size: 231.0 B, free: 1919.7 MiB)
17:38:10.771 INFO SparkContext - Created broadcast 399 from broadcast at BamSource.java:104
17:38:10.772 INFO MemoryStore - Block broadcast_400 stored as values in memory (estimated size 297.9 KiB, free 1918.5 MiB)
17:38:10.777 INFO MemoryStore - Block broadcast_400_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1918.5 MiB)
17:38:10.777 INFO BlockManagerInfo - Added broadcast_400_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.7 MiB)
17:38:10.778 INFO SparkContext - Created broadcast 400 from newAPIHadoopFile at PathSplitSource.java:96
17:38:10.786 INFO FileInputFormat - Total input files to process : 1
17:38:10.799 INFO SparkContext - Starting job: collect at ReadsSparkSinkUnitTest.java:182
17:38:10.800 INFO DAGScheduler - Got job 150 (collect at ReadsSparkSinkUnitTest.java:182) with 1 output partitions
17:38:10.800 INFO DAGScheduler - Final stage: ResultStage 202 (collect at ReadsSparkSinkUnitTest.java:182)
17:38:10.800 INFO DAGScheduler - Parents of final stage: List()
17:38:10.800 INFO DAGScheduler - Missing parents: List()
17:38:10.800 INFO DAGScheduler - Submitting ResultStage 202 (MapPartitionsRDD[966] at filter at ReadsSparkSource.java:96), which has no missing parents
17:38:10.805 INFO MemoryStore - Block broadcast_401 stored as values in memory (estimated size 148.2 KiB, free 1918.3 MiB)
17:38:10.806 INFO MemoryStore - Block broadcast_401_piece0 stored as bytes in memory (estimated size 54.6 KiB, free 1918.3 MiB)
17:38:10.806 INFO BlockManagerInfo - Added broadcast_401_piece0 in memory on localhost:37953 (size: 54.6 KiB, free: 1919.6 MiB)
17:38:10.806 INFO SparkContext - Created broadcast 401 from broadcast at DAGScheduler.scala:1580
17:38:10.806 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 202 (MapPartitionsRDD[966] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:38:10.806 INFO TaskSchedulerImpl - Adding task set 202.0 with 1 tasks resource profile 0
17:38:10.807 INFO TaskSetManager - Starting task 0.0 in stage 202.0 (TID 258) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7810 bytes)
17:38:10.807 INFO Executor - Running task 0.0 in stage 202.0 (TID 258)
17:38:10.819 INFO NewHadoopRDD - Input split: file:/tmp/ReadsSparkSinkUnitTest316721804858257364706.bam:0+237078
17:38:10.820 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:10.824 INFO Executor - Finished task 0.0 in stage 202.0 (TID 258). 749470 bytes result sent to driver
17:38:10.825 INFO TaskSetManager - Finished task 0.0 in stage 202.0 (TID 258) in 18 ms on localhost (executor driver) (1/1)
17:38:10.825 INFO TaskSchedulerImpl - Removed TaskSet 202.0, whose tasks have all completed, from pool
17:38:10.825 INFO DAGScheduler - ResultStage 202 (collect at ReadsSparkSinkUnitTest.java:182) finished in 0.025 s
17:38:10.825 INFO DAGScheduler - Job 150 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:10.825 INFO TaskSchedulerImpl - Killing all running tasks in stage 202: Stage finished
17:38:10.826 INFO DAGScheduler - Job 150 finished: collect at ReadsSparkSinkUnitTest.java:182, took 0.026270 s
17:38:10.837 INFO SparkContext - Starting job: count at ReadsSparkSinkUnitTest.java:185
17:38:10.837 INFO DAGScheduler - Got job 151 (count at ReadsSparkSinkUnitTest.java:185) with 1 output partitions
17:38:10.837 INFO DAGScheduler - Final stage: ResultStage 203 (count at ReadsSparkSinkUnitTest.java:185)
17:38:10.837 INFO DAGScheduler - Parents of final stage: List()
17:38:10.837 INFO DAGScheduler - Missing parents: List()
17:38:10.837 INFO DAGScheduler - Submitting ResultStage 203 (MapPartitionsRDD[948] at filter at ReadsSparkSource.java:96), which has no missing parents
17:38:10.853 INFO MemoryStore - Block broadcast_402 stored as values in memory (estimated size 426.1 KiB, free 1917.9 MiB)
17:38:10.857 INFO BlockManagerInfo - Removed broadcast_398_piece0 on localhost:37953 in memory (size: 58.5 KiB, free: 1919.7 MiB)
17:38:10.857 INFO MemoryStore - Block broadcast_402_piece0 stored as bytes in memory (estimated size 153.6 KiB, free 1917.9 MiB)
17:38:10.857 INFO BlockManagerInfo - Removed broadcast_396_piece0 on localhost:37953 in memory (size: 1890.0 B, free: 1919.7 MiB)
17:38:10.857 INFO BlockManagerInfo - Added broadcast_402_piece0 in memory on localhost:37953 (size: 153.6 KiB, free: 1919.5 MiB)
17:38:10.858 INFO SparkContext - Created broadcast 402 from broadcast at DAGScheduler.scala:1580
17:38:10.858 INFO BlockManagerInfo - Removed broadcast_395_piece0 on localhost:37953 in memory (size: 1890.0 B, free: 1919.5 MiB)
17:38:10.858 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 203 (MapPartitionsRDD[948] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:38:10.858 INFO TaskSchedulerImpl - Adding task set 203.0 with 1 tasks resource profile 0
17:38:10.858 INFO BlockManagerInfo - Removed broadcast_401_piece0 on localhost:37953 in memory (size: 54.6 KiB, free: 1919.6 MiB)
17:38:10.859 INFO BlockManagerInfo - Removed broadcast_397_piece0 on localhost:37953 in memory (size: 157.6 KiB, free: 1919.8 MiB)
17:38:10.859 INFO TaskSetManager - Starting task 0.0 in stage 203.0 (TID 259) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7893 bytes)
17:38:10.859 INFO Executor - Running task 0.0 in stage 203.0 (TID 259)
17:38:10.890 INFO NewHadoopRDD - Input split: file:/home/runner/work/gatk/gatk/src/test/resources/org/broadinstitute/hellbender/tools/BQSR/CEUTrio.HiSeq.WGS.b37.ch20.1m-1m1k.NA12878.bam:0+211123
17:38:10.891 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:10.892 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:10.897 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:10.900 INFO Executor - Finished task 0.0 in stage 203.0 (TID 259). 989 bytes result sent to driver
17:38:10.901 INFO TaskSetManager - Finished task 0.0 in stage 203.0 (TID 259) in 42 ms on localhost (executor driver) (1/1)
17:38:10.901 INFO TaskSchedulerImpl - Removed TaskSet 203.0, whose tasks have all completed, from pool
17:38:10.901 INFO DAGScheduler - ResultStage 203 (count at ReadsSparkSinkUnitTest.java:185) finished in 0.064 s
17:38:10.901 INFO DAGScheduler - Job 151 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:10.901 INFO TaskSchedulerImpl - Killing all running tasks in stage 203: Stage finished
17:38:10.901 INFO DAGScheduler - Job 151 finished: count at ReadsSparkSinkUnitTest.java:185, took 0.064621 s
17:38:10.904 INFO SparkContext - Starting job: count at ReadsSparkSinkUnitTest.java:185
17:38:10.905 INFO DAGScheduler - Got job 152 (count at ReadsSparkSinkUnitTest.java:185) with 1 output partitions
17:38:10.905 INFO DAGScheduler - Final stage: ResultStage 204 (count at ReadsSparkSinkUnitTest.java:185)
17:38:10.905 INFO DAGScheduler - Parents of final stage: List()
17:38:10.905 INFO DAGScheduler - Missing parents: List()
17:38:10.905 INFO DAGScheduler - Submitting ResultStage 204 (MapPartitionsRDD[966] at filter at ReadsSparkSource.java:96), which has no missing parents
17:38:10.910 INFO MemoryStore - Block broadcast_403 stored as values in memory (estimated size 148.1 KiB, free 1918.6 MiB)
17:38:10.911 INFO MemoryStore - Block broadcast_403_piece0 stored as bytes in memory (estimated size 54.5 KiB, free 1918.6 MiB)
17:38:10.911 INFO BlockManagerInfo - Added broadcast_403_piece0 in memory on localhost:37953 (size: 54.5 KiB, free: 1919.7 MiB)
17:38:10.911 INFO SparkContext - Created broadcast 403 from broadcast at DAGScheduler.scala:1580
17:38:10.911 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 204 (MapPartitionsRDD[966] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:38:10.911 INFO TaskSchedulerImpl - Adding task set 204.0 with 1 tasks resource profile 0
17:38:10.912 INFO TaskSetManager - Starting task 0.0 in stage 204.0 (TID 260) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7810 bytes)
17:38:10.912 INFO Executor - Running task 0.0 in stage 204.0 (TID 260)
17:38:10.923 INFO NewHadoopRDD - Input split: file:/tmp/ReadsSparkSinkUnitTest316721804858257364706.bam:0+237078
17:38:10.923 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:10.926 INFO Executor - Finished task 0.0 in stage 204.0 (TID 260). 989 bytes result sent to driver
17:38:10.926 INFO TaskSetManager - Finished task 0.0 in stage 204.0 (TID 260) in 14 ms on localhost (executor driver) (1/1)
17:38:10.926 INFO TaskSchedulerImpl - Removed TaskSet 204.0, whose tasks have all completed, from pool
17:38:10.927 INFO DAGScheduler - ResultStage 204 (count at ReadsSparkSinkUnitTest.java:185) finished in 0.022 s
17:38:10.927 INFO DAGScheduler - Job 152 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:10.927 INFO TaskSchedulerImpl - Killing all running tasks in stage 204: Stage finished
17:38:10.927 INFO DAGScheduler - Job 152 finished: count at ReadsSparkSinkUnitTest.java:185, took 0.022344 s
17:38:10.933 INFO MemoryStore - Block broadcast_404 stored as values in memory (estimated size 576.0 B, free 1918.6 MiB)
17:38:10.933 INFO MemoryStore - Block broadcast_404_piece0 stored as bytes in memory (estimated size 228.0 B, free 1918.6 MiB)
17:38:10.933 INFO BlockManagerInfo - Added broadcast_404_piece0 in memory on localhost:37953 (size: 228.0 B, free: 1919.7 MiB)
17:38:10.933 INFO SparkContext - Created broadcast 404 from broadcast at CramSource.java:114
17:38:10.934 INFO MemoryStore - Block broadcast_405 stored as values in memory (estimated size 297.9 KiB, free 1918.3 MiB)
17:38:10.940 INFO MemoryStore - Block broadcast_405_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1918.2 MiB)
17:38:10.940 INFO BlockManagerInfo - Added broadcast_405_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.6 MiB)
17:38:10.940 INFO SparkContext - Created broadcast 405 from newAPIHadoopFile at PathSplitSource.java:96
17:38:10.955 INFO MemoryStore - Block broadcast_406 stored as values in memory (estimated size 576.0 B, free 1918.2 MiB)
17:38:10.955 INFO MemoryStore - Block broadcast_406_piece0 stored as bytes in memory (estimated size 228.0 B, free 1918.2 MiB)
17:38:10.955 INFO BlockManagerInfo - Added broadcast_406_piece0 in memory on localhost:37953 (size: 228.0 B, free: 1919.6 MiB)
17:38:10.955 INFO SparkContext - Created broadcast 406 from broadcast at CramSource.java:114
17:38:10.956 INFO MemoryStore - Block broadcast_407 stored as values in memory (estimated size 297.9 KiB, free 1917.9 MiB)
17:38:10.962 INFO MemoryStore - Block broadcast_407_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1917.9 MiB)
17:38:10.962 INFO BlockManagerInfo - Added broadcast_407_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.6 MiB)
17:38:10.962 INFO SparkContext - Created broadcast 407 from newAPIHadoopFile at PathSplitSource.java:96
17:38:10.975 INFO FileInputFormat - Total input files to process : 1
17:38:10.976 INFO MemoryStore - Block broadcast_408 stored as values in memory (estimated size 6.0 KiB, free 1917.9 MiB)
17:38:10.980 INFO MemoryStore - Block broadcast_408_piece0 stored as bytes in memory (estimated size 1473.0 B, free 1917.9 MiB)
17:38:10.980 INFO BlockManagerInfo - Added broadcast_408_piece0 in memory on localhost:37953 (size: 1473.0 B, free: 1919.6 MiB)
17:38:10.980 INFO SparkContext - Created broadcast 408 from broadcast at ReadsSparkSink.java:133
17:38:10.980 INFO BlockManagerInfo - Removed broadcast_403_piece0 on localhost:37953 in memory (size: 54.5 KiB, free: 1919.7 MiB)
17:38:10.981 INFO BlockManagerInfo - Removed broadcast_402_piece0 on localhost:37953 in memory (size: 153.6 KiB, free: 1919.8 MiB)
17:38:10.981 INFO MemoryStore - Block broadcast_409 stored as values in memory (estimated size 6.2 KiB, free 1918.6 MiB)
17:38:10.981 INFO MemoryStore - Block broadcast_409_piece0 stored as bytes in memory (estimated size 1473.0 B, free 1919.0 MiB)
17:38:10.981 INFO BlockManagerInfo - Removed broadcast_393_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.9 MiB)
17:38:10.981 INFO BlockManagerInfo - Added broadcast_409_piece0 in memory on localhost:37953 (size: 1473.0 B, free: 1919.8 MiB)
17:38:10.982 INFO SparkContext - Created broadcast 409 from broadcast at CramSink.java:76
17:38:10.982 INFO BlockManagerInfo - Removed broadcast_399_piece0 on localhost:37953 in memory (size: 231.0 B, free: 1919.8 MiB)
17:38:10.982 INFO BlockManagerInfo - Removed broadcast_400_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.9 MiB)
17:38:10.983 INFO BlockManagerInfo - Removed broadcast_407_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.9 MiB)
17:38:10.983 INFO BlockManagerInfo - Removed broadcast_406_piece0 on localhost:37953 in memory (size: 228.0 B, free: 1919.9 MiB)
17:38:10.983 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:10.983 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:10.983 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:10.996 INFO SparkContext - Starting job: runJob at SparkHadoopWriter.scala:83
17:38:10.996 INFO DAGScheduler - Registering RDD 978 (mapToPair at SparkUtils.java:161) as input to shuffle 41
17:38:10.997 INFO DAGScheduler - Got job 153 (runJob at SparkHadoopWriter.scala:83) with 1 output partitions
17:38:10.997 INFO DAGScheduler - Final stage: ResultStage 206 (runJob at SparkHadoopWriter.scala:83)
17:38:10.997 INFO DAGScheduler - Parents of final stage: List(ShuffleMapStage 205)
17:38:10.997 INFO DAGScheduler - Missing parents: List(ShuffleMapStage 205)
17:38:10.997 INFO DAGScheduler - Submitting ShuffleMapStage 205 (MapPartitionsRDD[978] at mapToPair at SparkUtils.java:161), which has no missing parents
17:38:11.007 INFO MemoryStore - Block broadcast_410 stored as values in memory (estimated size 292.8 KiB, free 1919.4 MiB)
17:38:11.008 INFO MemoryStore - Block broadcast_410_piece0 stored as bytes in memory (estimated size 107.3 KiB, free 1919.3 MiB)
17:38:11.008 INFO BlockManagerInfo - Added broadcast_410_piece0 in memory on localhost:37953 (size: 107.3 KiB, free: 1919.8 MiB)
17:38:11.009 INFO SparkContext - Created broadcast 410 from broadcast at DAGScheduler.scala:1580
17:38:11.009 INFO DAGScheduler - Submitting 1 missing tasks from ShuffleMapStage 205 (MapPartitionsRDD[978] at mapToPair at SparkUtils.java:161) (first 15 tasks are for partitions Vector(0))
17:38:11.009 INFO TaskSchedulerImpl - Adding task set 205.0 with 1 tasks resource profile 0
17:38:11.009 INFO TaskSetManager - Starting task 0.0 in stage 205.0 (TID 261) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7869 bytes)
17:38:11.010 INFO Executor - Running task 0.0 in stage 205.0 (TID 261)
17:38:11.032 INFO NewHadoopRDD - Input split: file:/home/runner/work/gatk/gatk/src/test/resources/org/broadinstitute/hellbender/tools/BQSR/NA12878.chr17_69k_70k.dictFix.cram:0+50619
17:38:11.042 INFO Executor - Finished task 0.0 in stage 205.0 (TID 261). 1148 bytes result sent to driver
17:38:11.042 INFO TaskSetManager - Finished task 0.0 in stage 205.0 (TID 261) in 33 ms on localhost (executor driver) (1/1)
17:38:11.042 INFO TaskSchedulerImpl - Removed TaskSet 205.0, whose tasks have all completed, from pool
17:38:11.042 INFO DAGScheduler - ShuffleMapStage 205 (mapToPair at SparkUtils.java:161) finished in 0.045 s
17:38:11.042 INFO DAGScheduler - looking for newly runnable stages
17:38:11.043 INFO DAGScheduler - running: HashSet()
17:38:11.043 INFO DAGScheduler - waiting: HashSet(ResultStage 206)
17:38:11.043 INFO DAGScheduler - failed: HashSet()
17:38:11.043 INFO DAGScheduler - Submitting ResultStage 206 (MapPartitionsRDD[983] at mapToPair at CramSink.java:89), which has no missing parents
17:38:11.049 INFO MemoryStore - Block broadcast_411 stored as values in memory (estimated size 153.2 KiB, free 1919.1 MiB)
17:38:11.049 INFO MemoryStore - Block broadcast_411_piece0 stored as bytes in memory (estimated size 58.1 KiB, free 1919.0 MiB)
17:38:11.049 INFO BlockManagerInfo - Added broadcast_411_piece0 in memory on localhost:37953 (size: 58.1 KiB, free: 1919.8 MiB)
17:38:11.050 INFO SparkContext - Created broadcast 411 from broadcast at DAGScheduler.scala:1580
17:38:11.050 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 206 (MapPartitionsRDD[983] at mapToPair at CramSink.java:89) (first 15 tasks are for partitions Vector(0))
17:38:11.050 INFO TaskSchedulerImpl - Adding task set 206.0 with 1 tasks resource profile 0
17:38:11.050 INFO TaskSetManager - Starting task 0.0 in stage 206.0 (TID 262) (localhost, executor driver, partition 0, NODE_LOCAL, 7513 bytes)
17:38:11.051 INFO Executor - Running task 0.0 in stage 206.0 (TID 262)
17:38:11.055 INFO ShuffleBlockFetcherIterator - Getting 1 (82.3 KiB) non-empty blocks including 1 (82.3 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
17:38:11.055 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
17:38:11.062 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:11.062 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:11.062 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:11.062 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:11.062 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:11.062 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:11.094 INFO FileOutputCommitter - Saved output of task 'attempt_202510271738105563288538512262268_0983_r_000000_0' to file:/tmp/ReadsSparkSinkUnitTest5.someOtherPlace9880284926956200687/_temporary/0/task_202510271738105563288538512262268_0983_r_000000
17:38:11.094 INFO SparkHadoopMapRedUtil - attempt_202510271738105563288538512262268_0983_r_000000_0: Committed. Elapsed time: 0 ms.
17:38:11.095 INFO Executor - Finished task 0.0 in stage 206.0 (TID 262). 1858 bytes result sent to driver
17:38:11.095 INFO TaskSetManager - Finished task 0.0 in stage 206.0 (TID 262) in 45 ms on localhost (executor driver) (1/1)
17:38:11.095 INFO TaskSchedulerImpl - Removed TaskSet 206.0, whose tasks have all completed, from pool
17:38:11.096 INFO DAGScheduler - ResultStage 206 (runJob at SparkHadoopWriter.scala:83) finished in 0.053 s
17:38:11.096 INFO DAGScheduler - Job 153 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:11.096 INFO TaskSchedulerImpl - Killing all running tasks in stage 206: Stage finished
17:38:11.096 INFO DAGScheduler - Job 153 finished: runJob at SparkHadoopWriter.scala:83, took 0.099668 s
17:38:11.096 INFO SparkHadoopWriter - Start to commit write Job job_202510271738105563288538512262268_0983.
17:38:11.100 INFO SparkHadoopWriter - Write Job job_202510271738105563288538512262268_0983 committed. Elapsed time: 3 ms.
17:38:11.108 INFO HadoopFileSystemWrapper - Concatenating 3 parts to /tmp/ReadsSparkSinkUnitTest516009963358806416673.cram
17:38:11.111 INFO HadoopFileSystemWrapper - Concatenating to /tmp/ReadsSparkSinkUnitTest516009963358806416673.cram done
17:38:11.113 INFO MemoryStore - Block broadcast_412 stored as values in memory (estimated size 504.0 B, free 1919.0 MiB)
17:38:11.113 INFO MemoryStore - Block broadcast_412_piece0 stored as bytes in memory (estimated size 160.0 B, free 1919.0 MiB)
17:38:11.113 INFO BlockManagerInfo - Added broadcast_412_piece0 in memory on localhost:37953 (size: 160.0 B, free: 1919.8 MiB)
17:38:11.113 INFO SparkContext - Created broadcast 412 from broadcast at CramSource.java:114
17:38:11.114 INFO MemoryStore - Block broadcast_413 stored as values in memory (estimated size 297.9 KiB, free 1918.8 MiB)
17:38:11.120 INFO MemoryStore - Block broadcast_413_piece0 stored as bytes in memory (estimated size 50.1 KiB, free 1918.7 MiB)
17:38:11.120 INFO BlockManagerInfo - Added broadcast_413_piece0 in memory on localhost:37953 (size: 50.1 KiB, free: 1919.7 MiB)
17:38:11.120 INFO SparkContext - Created broadcast 413 from newAPIHadoopFile at PathSplitSource.java:96
17:38:11.134 INFO FileInputFormat - Total input files to process : 1
17:38:11.156 INFO SparkContext - Starting job: collect at ReadsSparkSinkUnitTest.java:182
17:38:11.156 INFO DAGScheduler - Got job 154 (collect at ReadsSparkSinkUnitTest.java:182) with 1 output partitions
17:38:11.156 INFO DAGScheduler - Final stage: ResultStage 207 (collect at ReadsSparkSinkUnitTest.java:182)
17:38:11.156 INFO DAGScheduler - Parents of final stage: List()
17:38:11.157 INFO DAGScheduler - Missing parents: List()
17:38:11.157 INFO DAGScheduler - Submitting ResultStage 207 (MapPartitionsRDD[989] at filter at ReadsSparkSource.java:96), which has no missing parents
17:38:11.167 INFO MemoryStore - Block broadcast_414 stored as values in memory (estimated size 286.8 KiB, free 1918.4 MiB)
17:38:11.171 INFO BlockManagerInfo - Removed broadcast_409_piece0 on localhost:37953 in memory (size: 1473.0 B, free: 1919.7 MiB)
17:38:11.171 INFO MemoryStore - Block broadcast_414_piece0 stored as bytes in memory (estimated size 103.6 KiB, free 1918.3 MiB)
17:38:11.171 INFO BlockManagerInfo - Added broadcast_414_piece0 in memory on localhost:37953 (size: 103.6 KiB, free: 1919.6 MiB)
17:38:11.171 INFO SparkContext - Created broadcast 414 from broadcast at DAGScheduler.scala:1580
17:38:11.172 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 207 (MapPartitionsRDD[989] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:38:11.172 INFO TaskSchedulerImpl - Adding task set 207.0 with 1 tasks resource profile 0
17:38:11.172 INFO BlockManagerInfo - Removed broadcast_408_piece0 on localhost:37953 in memory (size: 1473.0 B, free: 1919.6 MiB)
17:38:11.172 INFO TaskSetManager - Starting task 0.0 in stage 207.0 (TID 263) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7811 bytes)
17:38:11.173 INFO BlockManagerInfo - Removed broadcast_410_piece0 on localhost:37953 in memory (size: 107.3 KiB, free: 1919.7 MiB)
17:38:11.173 INFO Executor - Running task 0.0 in stage 207.0 (TID 263)
17:38:11.173 INFO BlockManagerInfo - Removed broadcast_411_piece0 on localhost:37953 in memory (size: 58.1 KiB, free: 1919.8 MiB)
17:38:11.195 INFO NewHadoopRDD - Input split: file:/tmp/ReadsSparkSinkUnitTest516009963358806416673.cram:0+43713
17:38:11.205 INFO Executor - Finished task 0.0 in stage 207.0 (TID 263). 154058 bytes result sent to driver
17:38:11.205 INFO TaskSetManager - Finished task 0.0 in stage 207.0 (TID 263) in 33 ms on localhost (executor driver) (1/1)
17:38:11.205 INFO TaskSchedulerImpl - Removed TaskSet 207.0, whose tasks have all completed, from pool
17:38:11.206 INFO DAGScheduler - ResultStage 207 (collect at ReadsSparkSinkUnitTest.java:182) finished in 0.048 s
17:38:11.206 INFO DAGScheduler - Job 154 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:11.206 INFO TaskSchedulerImpl - Killing all running tasks in stage 207: Stage finished
17:38:11.206 INFO DAGScheduler - Job 154 finished: collect at ReadsSparkSinkUnitTest.java:182, took 0.049553 s
17:38:11.211 INFO SparkContext - Starting job: count at ReadsSparkSinkUnitTest.java:185
17:38:11.211 INFO DAGScheduler - Got job 155 (count at ReadsSparkSinkUnitTest.java:185) with 1 output partitions
17:38:11.211 INFO DAGScheduler - Final stage: ResultStage 208 (count at ReadsSparkSinkUnitTest.java:185)
17:38:11.211 INFO DAGScheduler - Parents of final stage: List()
17:38:11.211 INFO DAGScheduler - Missing parents: List()
17:38:11.211 INFO DAGScheduler - Submitting ResultStage 208 (MapPartitionsRDD[972] at filter at ReadsSparkSource.java:96), which has no missing parents
17:38:11.221 INFO MemoryStore - Block broadcast_415 stored as values in memory (estimated size 286.8 KiB, free 1918.7 MiB)
17:38:11.222 INFO MemoryStore - Block broadcast_415_piece0 stored as bytes in memory (estimated size 103.6 KiB, free 1918.6 MiB)
17:38:11.222 INFO BlockManagerInfo - Added broadcast_415_piece0 in memory on localhost:37953 (size: 103.6 KiB, free: 1919.7 MiB)
17:38:11.223 INFO SparkContext - Created broadcast 415 from broadcast at DAGScheduler.scala:1580
17:38:11.223 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 208 (MapPartitionsRDD[972] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:38:11.223 INFO TaskSchedulerImpl - Adding task set 208.0 with 1 tasks resource profile 0
17:38:11.223 INFO TaskSetManager - Starting task 0.0 in stage 208.0 (TID 264) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7880 bytes)
17:38:11.223 INFO Executor - Running task 0.0 in stage 208.0 (TID 264)
17:38:11.244 INFO NewHadoopRDD - Input split: file:/home/runner/work/gatk/gatk/src/test/resources/org/broadinstitute/hellbender/tools/BQSR/NA12878.chr17_69k_70k.dictFix.cram:0+50619
17:38:11.250 INFO Executor - Finished task 0.0 in stage 208.0 (TID 264). 989 bytes result sent to driver
17:38:11.250 INFO TaskSetManager - Finished task 0.0 in stage 208.0 (TID 264) in 27 ms on localhost (executor driver) (1/1)
17:38:11.250 INFO TaskSchedulerImpl - Removed TaskSet 208.0, whose tasks have all completed, from pool
17:38:11.250 INFO DAGScheduler - ResultStage 208 (count at ReadsSparkSinkUnitTest.java:185) finished in 0.039 s
17:38:11.250 INFO DAGScheduler - Job 155 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:11.250 INFO TaskSchedulerImpl - Killing all running tasks in stage 208: Stage finished
17:38:11.251 INFO DAGScheduler - Job 155 finished: count at ReadsSparkSinkUnitTest.java:185, took 0.039981 s
17:38:11.254 INFO SparkContext - Starting job: count at ReadsSparkSinkUnitTest.java:185
17:38:11.254 INFO DAGScheduler - Got job 156 (count at ReadsSparkSinkUnitTest.java:185) with 1 output partitions
17:38:11.254 INFO DAGScheduler - Final stage: ResultStage 209 (count at ReadsSparkSinkUnitTest.java:185)
17:38:11.254 INFO DAGScheduler - Parents of final stage: List()
17:38:11.254 INFO DAGScheduler - Missing parents: List()
17:38:11.254 INFO DAGScheduler - Submitting ResultStage 209 (MapPartitionsRDD[989] at filter at ReadsSparkSource.java:96), which has no missing parents
17:38:11.264 INFO MemoryStore - Block broadcast_416 stored as values in memory (estimated size 286.8 KiB, free 1918.3 MiB)
17:38:11.265 INFO MemoryStore - Block broadcast_416_piece0 stored as bytes in memory (estimated size 103.6 KiB, free 1918.2 MiB)
17:38:11.265 INFO BlockManagerInfo - Added broadcast_416_piece0 in memory on localhost:37953 (size: 103.6 KiB, free: 1919.6 MiB)
17:38:11.265 INFO SparkContext - Created broadcast 416 from broadcast at DAGScheduler.scala:1580
17:38:11.265 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 209 (MapPartitionsRDD[989] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:38:11.265 INFO TaskSchedulerImpl - Adding task set 209.0 with 1 tasks resource profile 0
17:38:11.266 INFO TaskSetManager - Starting task 0.0 in stage 209.0 (TID 265) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7811 bytes)
17:38:11.266 INFO Executor - Running task 0.0 in stage 209.0 (TID 265)
17:38:11.286 INFO NewHadoopRDD - Input split: file:/tmp/ReadsSparkSinkUnitTest516009963358806416673.cram:0+43713
17:38:11.295 INFO Executor - Finished task 0.0 in stage 209.0 (TID 265). 989 bytes result sent to driver
17:38:11.296 INFO TaskSetManager - Finished task 0.0 in stage 209.0 (TID 265) in 30 ms on localhost (executor driver) (1/1)
17:38:11.296 INFO TaskSchedulerImpl - Removed TaskSet 209.0, whose tasks have all completed, from pool
17:38:11.296 INFO DAGScheduler - ResultStage 209 (count at ReadsSparkSinkUnitTest.java:185) finished in 0.042 s
17:38:11.296 INFO DAGScheduler - Job 156 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:11.296 INFO TaskSchedulerImpl - Killing all running tasks in stage 209: Stage finished
17:38:11.296 INFO DAGScheduler - Job 156 finished: count at ReadsSparkSinkUnitTest.java:185, took 0.042340 s
17:38:11.300 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:11.302 INFO MemoryStore - Block broadcast_417 stored as values in memory (estimated size 297.9 KiB, free 1917.9 MiB)
17:38:11.308 INFO MemoryStore - Block broadcast_417_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1917.8 MiB)
17:38:11.308 INFO BlockManagerInfo - Added broadcast_417_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.5 MiB)
17:38:11.308 INFO SparkContext - Created broadcast 417 from newAPIHadoopFile at PathSplitSource.java:96
17:38:11.327 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:11.328 INFO MemoryStore - Block broadcast_418 stored as values in memory (estimated size 297.9 KiB, free 1917.5 MiB)
17:38:11.334 INFO MemoryStore - Block broadcast_418_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1917.5 MiB)
17:38:11.334 INFO BlockManagerInfo - Added broadcast_418_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.5 MiB)
17:38:11.334 INFO SparkContext - Created broadcast 418 from newAPIHadoopFile at PathSplitSource.java:96
17:38:11.353 INFO FileInputFormat - Total input files to process : 1
17:38:11.354 INFO MemoryStore - Block broadcast_419 stored as values in memory (estimated size 160.7 KiB, free 1917.3 MiB)
17:38:11.355 INFO MemoryStore - Block broadcast_419_piece0 stored as bytes in memory (estimated size 9.6 KiB, free 1917.3 MiB)
17:38:11.355 INFO BlockManagerInfo - Added broadcast_419_piece0 in memory on localhost:37953 (size: 9.6 KiB, free: 1919.5 MiB)
17:38:11.355 INFO SparkContext - Created broadcast 419 from broadcast at ReadsSparkSink.java:133
17:38:11.360 INFO HadoopMapRedCommitProtocol - Using output committer class org.apache.hadoop.mapred.FileOutputCommitter
17:38:11.360 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:11.360 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:11.372 INFO SparkContext - Starting job: runJob at SparkHadoopWriter.scala:83
17:38:11.372 INFO DAGScheduler - Registering RDD 1003 (mapToPair at SparkUtils.java:161) as input to shuffle 42
17:38:11.373 INFO DAGScheduler - Got job 157 (runJob at SparkHadoopWriter.scala:83) with 1 output partitions
17:38:11.373 INFO DAGScheduler - Final stage: ResultStage 211 (runJob at SparkHadoopWriter.scala:83)
17:38:11.373 INFO DAGScheduler - Parents of final stage: List(ShuffleMapStage 210)
17:38:11.373 INFO DAGScheduler - Missing parents: List(ShuffleMapStage 210)
17:38:11.373 INFO DAGScheduler - Submitting ShuffleMapStage 210 (MapPartitionsRDD[1003] at mapToPair at SparkUtils.java:161), which has no missing parents
17:38:11.388 INFO MemoryStore - Block broadcast_420 stored as values in memory (estimated size 520.4 KiB, free 1916.8 MiB)
17:38:11.392 INFO BlockManagerInfo - Removed broadcast_416_piece0 on localhost:37953 in memory (size: 103.6 KiB, free: 1919.6 MiB)
17:38:11.392 INFO BlockManagerInfo - Removed broadcast_404_piece0 on localhost:37953 in memory (size: 228.0 B, free: 1919.6 MiB)
17:38:11.392 INFO MemoryStore - Block broadcast_420_piece0 stored as bytes in memory (estimated size 166.1 KiB, free 1917.0 MiB)
17:38:11.393 INFO BlockManagerInfo - Added broadcast_420_piece0 in memory on localhost:37953 (size: 166.1 KiB, free: 1919.4 MiB)
17:38:11.393 INFO BlockManagerInfo - Removed broadcast_412_piece0 on localhost:37953 in memory (size: 160.0 B, free: 1919.4 MiB)
17:38:11.393 INFO SparkContext - Created broadcast 420 from broadcast at DAGScheduler.scala:1580
17:38:11.393 INFO DAGScheduler - Submitting 1 missing tasks from ShuffleMapStage 210 (MapPartitionsRDD[1003] at mapToPair at SparkUtils.java:161) (first 15 tasks are for partitions Vector(0))
17:38:11.393 INFO TaskSchedulerImpl - Adding task set 210.0 with 1 tasks resource profile 0
17:38:11.393 INFO BlockManagerInfo - Removed broadcast_418_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.5 MiB)
17:38:11.394 INFO BlockManagerInfo - Removed broadcast_413_piece0 on localhost:37953 in memory (size: 50.1 KiB, free: 1919.5 MiB)
17:38:11.394 INFO TaskSetManager - Starting task 0.0 in stage 210.0 (TID 266) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7861 bytes)
17:38:11.394 INFO BlockManagerInfo - Removed broadcast_414_piece0 on localhost:37953 in memory (size: 103.6 KiB, free: 1919.6 MiB)
17:38:11.394 INFO Executor - Running task 0.0 in stage 210.0 (TID 266)
17:38:11.396 INFO BlockManagerInfo - Removed broadcast_415_piece0 on localhost:37953 in memory (size: 103.6 KiB, free: 1919.7 MiB)
17:38:11.396 INFO BlockManagerInfo - Removed broadcast_405_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.8 MiB)
17:38:11.426 INFO NewHadoopRDD - Input split: file:/home/runner/work/gatk/gatk/src/test/resources/org/broadinstitute/hellbender/tools/BQSR/HiSeq.1mb.1RG.2k_lines.bam:0+222075
17:38:11.427 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:11.428 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:11.435 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:11.443 INFO Executor - Finished task 0.0 in stage 210.0 (TID 266). 1148 bytes result sent to driver
17:38:11.443 INFO TaskSetManager - Finished task 0.0 in stage 210.0 (TID 266) in 49 ms on localhost (executor driver) (1/1)
17:38:11.443 INFO TaskSchedulerImpl - Removed TaskSet 210.0, whose tasks have all completed, from pool
17:38:11.444 INFO DAGScheduler - ShuffleMapStage 210 (mapToPair at SparkUtils.java:161) finished in 0.070 s
17:38:11.444 INFO DAGScheduler - looking for newly runnable stages
17:38:11.444 INFO DAGScheduler - running: HashSet()
17:38:11.444 INFO DAGScheduler - waiting: HashSet(ResultStage 211)
17:38:11.444 INFO DAGScheduler - failed: HashSet()
17:38:11.444 INFO DAGScheduler - Submitting ResultStage 211 (MapPartitionsRDD[1009] at saveAsTextFile at SamSink.java:65), which has no missing parents
17:38:11.450 INFO MemoryStore - Block broadcast_421 stored as values in memory (estimated size 241.1 KiB, free 1918.6 MiB)
17:38:11.450 INFO MemoryStore - Block broadcast_421_piece0 stored as bytes in memory (estimated size 66.9 KiB, free 1918.5 MiB)
17:38:11.450 INFO BlockManagerInfo - Added broadcast_421_piece0 in memory on localhost:37953 (size: 66.9 KiB, free: 1919.7 MiB)
17:38:11.451 INFO SparkContext - Created broadcast 421 from broadcast at DAGScheduler.scala:1580
17:38:11.451 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 211 (MapPartitionsRDD[1009] at saveAsTextFile at SamSink.java:65) (first 15 tasks are for partitions Vector(0))
17:38:11.451 INFO TaskSchedulerImpl - Adding task set 211.0 with 1 tasks resource profile 0
17:38:11.451 INFO TaskSetManager - Starting task 0.0 in stage 211.0 (TID 267) (localhost, executor driver, partition 0, NODE_LOCAL, 7513 bytes)
17:38:11.452 INFO Executor - Running task 0.0 in stage 211.0 (TID 267)
17:38:11.456 INFO ShuffleBlockFetcherIterator - Getting 1 (343.8 KiB) non-empty blocks including 1 (343.8 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
17:38:11.456 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
17:38:11.467 INFO HadoopMapRedCommitProtocol - Using output committer class org.apache.hadoop.mapred.FileOutputCommitter
17:38:11.467 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:11.467 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:11.481 INFO FileOutputCommitter - Saved output of task 'attempt_202510271738118109778778305718545_1009_m_000000_0' to file:/tmp/ReadsSparkSinkUnitTest6.someOtherPlace8850484837185718135/_temporary/0/task_202510271738118109778778305718545_1009_m_000000
17:38:11.481 INFO SparkHadoopMapRedUtil - attempt_202510271738118109778778305718545_1009_m_000000_0: Committed. Elapsed time: 0 ms.
17:38:11.482 INFO Executor - Finished task 0.0 in stage 211.0 (TID 267). 1858 bytes result sent to driver
17:38:11.482 INFO TaskSetManager - Finished task 0.0 in stage 211.0 (TID 267) in 31 ms on localhost (executor driver) (1/1)
17:38:11.482 INFO TaskSchedulerImpl - Removed TaskSet 211.0, whose tasks have all completed, from pool
17:38:11.482 INFO DAGScheduler - ResultStage 211 (runJob at SparkHadoopWriter.scala:83) finished in 0.038 s
17:38:11.482 INFO DAGScheduler - Job 157 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:11.482 INFO TaskSchedulerImpl - Killing all running tasks in stage 211: Stage finished
17:38:11.482 INFO DAGScheduler - Job 157 finished: runJob at SparkHadoopWriter.scala:83, took 0.110441 s
17:38:11.483 INFO SparkHadoopWriter - Start to commit write Job job_202510271738118109778778305718545_1009.
17:38:11.486 INFO SparkHadoopWriter - Write Job job_202510271738118109778778305718545_1009 committed. Elapsed time: 3 ms.
17:38:11.491 INFO HadoopFileSystemWrapper - Concatenating 2 parts to /tmp/ReadsSparkSinkUnitTest66295412472430740802.sam
17:38:11.495 INFO HadoopFileSystemWrapper - Concatenating to /tmp/ReadsSparkSinkUnitTest66295412472430740802.sam done
WARNING 2025-10-27 17:38:11 SamReaderFactory Unable to detect file format from input URL or stream, assuming SAM format.
WARNING 2025-10-27 17:38:11 SamReaderFactory Unable to detect file format from input URL or stream, assuming SAM format.
17:38:11.497 INFO MemoryStore - Block broadcast_422 stored as values in memory (estimated size 160.7 KiB, free 1918.4 MiB)
17:38:11.498 INFO MemoryStore - Block broadcast_422_piece0 stored as bytes in memory (estimated size 9.6 KiB, free 1918.4 MiB)
17:38:11.498 INFO BlockManagerInfo - Added broadcast_422_piece0 in memory on localhost:37953 (size: 9.6 KiB, free: 1919.7 MiB)
17:38:11.498 INFO SparkContext - Created broadcast 422 from broadcast at SamSource.java:78
17:38:11.499 INFO MemoryStore - Block broadcast_423 stored as values in memory (estimated size 297.9 KiB, free 1918.1 MiB)
17:38:11.505 INFO MemoryStore - Block broadcast_423_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1918.0 MiB)
17:38:11.505 INFO BlockManagerInfo - Added broadcast_423_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.7 MiB)
17:38:11.505 INFO SparkContext - Created broadcast 423 from newAPIHadoopFile at SamSource.java:108
17:38:11.508 INFO FileInputFormat - Total input files to process : 1
17:38:11.511 INFO SparkContext - Starting job: collect at ReadsSparkSinkUnitTest.java:182
17:38:11.511 INFO DAGScheduler - Got job 158 (collect at ReadsSparkSinkUnitTest.java:182) with 1 output partitions
17:38:11.511 INFO DAGScheduler - Final stage: ResultStage 212 (collect at ReadsSparkSinkUnitTest.java:182)
17:38:11.511 INFO DAGScheduler - Parents of final stage: List()
17:38:11.511 INFO DAGScheduler - Missing parents: List()
17:38:11.511 INFO DAGScheduler - Submitting ResultStage 212 (MapPartitionsRDD[1014] at filter at ReadsSparkSource.java:96), which has no missing parents
17:38:11.512 INFO MemoryStore - Block broadcast_424 stored as values in memory (estimated size 7.5 KiB, free 1918.0 MiB)
17:38:11.512 INFO MemoryStore - Block broadcast_424_piece0 stored as bytes in memory (estimated size 3.8 KiB, free 1918.0 MiB)
17:38:11.512 INFO BlockManagerInfo - Added broadcast_424_piece0 in memory on localhost:37953 (size: 3.8 KiB, free: 1919.7 MiB)
17:38:11.512 INFO SparkContext - Created broadcast 424 from broadcast at DAGScheduler.scala:1580
17:38:11.512 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 212 (MapPartitionsRDD[1014] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:38:11.513 INFO TaskSchedulerImpl - Adding task set 212.0 with 1 tasks resource profile 0
17:38:11.513 INFO TaskSetManager - Starting task 0.0 in stage 212.0 (TID 268) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7809 bytes)
17:38:11.513 INFO Executor - Running task 0.0 in stage 212.0 (TID 268)
17:38:11.514 INFO NewHadoopRDD - Input split: file:/tmp/ReadsSparkSinkUnitTest66295412472430740802.sam:0+847558
17:38:11.524 INFO Executor - Finished task 0.0 in stage 212.0 (TID 268). 651483 bytes result sent to driver
17:38:11.526 INFO TaskSetManager - Finished task 0.0 in stage 212.0 (TID 268) in 13 ms on localhost (executor driver) (1/1)
17:38:11.526 INFO TaskSchedulerImpl - Removed TaskSet 212.0, whose tasks have all completed, from pool
17:38:11.526 INFO DAGScheduler - ResultStage 212 (collect at ReadsSparkSinkUnitTest.java:182) finished in 0.015 s
17:38:11.526 INFO DAGScheduler - Job 158 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:11.526 INFO TaskSchedulerImpl - Killing all running tasks in stage 212: Stage finished
17:38:11.526 INFO DAGScheduler - Job 158 finished: collect at ReadsSparkSinkUnitTest.java:182, took 0.015183 s
17:38:11.536 INFO SparkContext - Starting job: count at ReadsSparkSinkUnitTest.java:185
17:38:11.536 INFO DAGScheduler - Got job 159 (count at ReadsSparkSinkUnitTest.java:185) with 1 output partitions
17:38:11.536 INFO DAGScheduler - Final stage: ResultStage 213 (count at ReadsSparkSinkUnitTest.java:185)
17:38:11.536 INFO DAGScheduler - Parents of final stage: List()
17:38:11.536 INFO DAGScheduler - Missing parents: List()
17:38:11.536 INFO DAGScheduler - Submitting ResultStage 213 (MapPartitionsRDD[996] at filter at ReadsSparkSource.java:96), which has no missing parents
17:38:11.551 INFO MemoryStore - Block broadcast_425 stored as values in memory (estimated size 426.1 KiB, free 1917.6 MiB)
17:38:11.552 INFO MemoryStore - Block broadcast_425_piece0 stored as bytes in memory (estimated size 153.6 KiB, free 1917.4 MiB)
17:38:11.552 INFO BlockManagerInfo - Added broadcast_425_piece0 in memory on localhost:37953 (size: 153.6 KiB, free: 1919.5 MiB)
17:38:11.553 INFO SparkContext - Created broadcast 425 from broadcast at DAGScheduler.scala:1580
17:38:11.553 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 213 (MapPartitionsRDD[996] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:38:11.553 INFO TaskSchedulerImpl - Adding task set 213.0 with 1 tasks resource profile 0
17:38:11.553 INFO TaskSetManager - Starting task 0.0 in stage 213.0 (TID 269) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7872 bytes)
17:38:11.553 INFO Executor - Running task 0.0 in stage 213.0 (TID 269)
17:38:11.584 INFO NewHadoopRDD - Input split: file:/home/runner/work/gatk/gatk/src/test/resources/org/broadinstitute/hellbender/tools/BQSR/HiSeq.1mb.1RG.2k_lines.bam:0+222075
17:38:11.585 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:11.586 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:11.593 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:11.603 INFO Executor - Finished task 0.0 in stage 213.0 (TID 269). 1075 bytes result sent to driver
17:38:11.604 INFO TaskSetManager - Finished task 0.0 in stage 213.0 (TID 269) in 51 ms on localhost (executor driver) (1/1)
17:38:11.604 INFO TaskSchedulerImpl - Removed TaskSet 213.0, whose tasks have all completed, from pool
17:38:11.604 INFO DAGScheduler - ResultStage 213 (count at ReadsSparkSinkUnitTest.java:185) finished in 0.068 s
17:38:11.604 INFO DAGScheduler - Job 159 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:11.604 INFO TaskSchedulerImpl - Killing all running tasks in stage 213: Stage finished
17:38:11.604 INFO DAGScheduler - Job 159 finished: count at ReadsSparkSinkUnitTest.java:185, took 0.068441 s
17:38:11.605 INFO BlockManagerInfo - Removed broadcast_419_piece0 on localhost:37953 in memory (size: 9.6 KiB, free: 1919.5 MiB)
17:38:11.605 INFO BlockManagerInfo - Removed broadcast_420_piece0 on localhost:37953 in memory (size: 166.1 KiB, free: 1919.7 MiB)
17:38:11.605 INFO BlockManagerInfo - Removed broadcast_424_piece0 on localhost:37953 in memory (size: 3.8 KiB, free: 1919.7 MiB)
17:38:11.606 INFO BlockManagerInfo - Removed broadcast_421_piece0 on localhost:37953 in memory (size: 66.9 KiB, free: 1919.7 MiB)
17:38:11.607 INFO SparkContext - Starting job: count at ReadsSparkSinkUnitTest.java:185
17:38:11.607 INFO DAGScheduler - Got job 160 (count at ReadsSparkSinkUnitTest.java:185) with 1 output partitions
17:38:11.607 INFO DAGScheduler - Final stage: ResultStage 214 (count at ReadsSparkSinkUnitTest.java:185)
17:38:11.607 INFO DAGScheduler - Parents of final stage: List()
17:38:11.607 INFO DAGScheduler - Missing parents: List()
17:38:11.608 INFO DAGScheduler - Submitting ResultStage 214 (MapPartitionsRDD[1014] at filter at ReadsSparkSource.java:96), which has no missing parents
17:38:11.608 INFO MemoryStore - Block broadcast_426 stored as values in memory (estimated size 7.4 KiB, free 1918.6 MiB)
17:38:11.608 INFO MemoryStore - Block broadcast_426_piece0 stored as bytes in memory (estimated size 3.8 KiB, free 1918.6 MiB)
17:38:11.608 INFO BlockManagerInfo - Added broadcast_426_piece0 in memory on localhost:37953 (size: 3.8 KiB, free: 1919.7 MiB)
17:38:11.609 INFO SparkContext - Created broadcast 426 from broadcast at DAGScheduler.scala:1580
17:38:11.609 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 214 (MapPartitionsRDD[1014] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:38:11.609 INFO TaskSchedulerImpl - Adding task set 214.0 with 1 tasks resource profile 0
17:38:11.609 INFO TaskSetManager - Starting task 0.0 in stage 214.0 (TID 270) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7809 bytes)
17:38:11.610 INFO Executor - Running task 0.0 in stage 214.0 (TID 270)
17:38:11.611 INFO NewHadoopRDD - Input split: file:/tmp/ReadsSparkSinkUnitTest66295412472430740802.sam:0+847558
17:38:11.618 INFO Executor - Finished task 0.0 in stage 214.0 (TID 270). 946 bytes result sent to driver
17:38:11.618 INFO TaskSetManager - Finished task 0.0 in stage 214.0 (TID 270) in 9 ms on localhost (executor driver) (1/1)
17:38:11.618 INFO TaskSchedulerImpl - Removed TaskSet 214.0, whose tasks have all completed, from pool
17:38:11.618 INFO DAGScheduler - ResultStage 214 (count at ReadsSparkSinkUnitTest.java:185) finished in 0.010 s
17:38:11.618 INFO DAGScheduler - Job 160 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:11.618 INFO TaskSchedulerImpl - Killing all running tasks in stage 214: Stage finished
17:38:11.618 INFO DAGScheduler - Job 160 finished: count at ReadsSparkSinkUnitTest.java:185, took 0.011067 s
17:38:11.630 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam dst=null perm=null proto=rpc
17:38:11.631 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:11.632 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:38:11.632 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam dst=null perm=null proto=rpc
17:38:11.633 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:11.635 INFO MemoryStore - Block broadcast_427 stored as values in memory (estimated size 297.9 KiB, free 1918.3 MiB)
17:38:11.640 INFO MemoryStore - Block broadcast_427_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1918.2 MiB)
17:38:11.640 INFO BlockManagerInfo - Added broadcast_427_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.7 MiB)
17:38:11.641 INFO SparkContext - Created broadcast 427 from newAPIHadoopFile at PathSplitSource.java:96
17:38:11.659 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:11.661 INFO MemoryStore - Block broadcast_428 stored as values in memory (estimated size 297.9 KiB, free 1917.9 MiB)
17:38:11.667 INFO MemoryStore - Block broadcast_428_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1917.9 MiB)
17:38:11.667 INFO BlockManagerInfo - Added broadcast_428_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.6 MiB)
17:38:11.667 INFO SparkContext - Created broadcast 428 from newAPIHadoopFile at PathSplitSource.java:96
17:38:11.686 INFO FileInputFormat - Total input files to process : 1
17:38:11.688 INFO MemoryStore - Block broadcast_429 stored as values in memory (estimated size 160.7 KiB, free 1917.7 MiB)
17:38:11.689 INFO MemoryStore - Block broadcast_429_piece0 stored as bytes in memory (estimated size 9.6 KiB, free 1917.7 MiB)
17:38:11.689 INFO BlockManagerInfo - Added broadcast_429_piece0 in memory on localhost:37953 (size: 9.6 KiB, free: 1919.6 MiB)
17:38:11.689 INFO SparkContext - Created broadcast 429 from broadcast at ReadsSparkSink.java:133
17:38:11.690 INFO MemoryStore - Block broadcast_430 stored as values in memory (estimated size 163.2 KiB, free 1917.6 MiB)
17:38:11.691 INFO MemoryStore - Block broadcast_430_piece0 stored as bytes in memory (estimated size 9.6 KiB, free 1917.6 MiB)
17:38:11.691 INFO BlockManagerInfo - Added broadcast_430_piece0 in memory on localhost:37953 (size: 9.6 KiB, free: 1919.6 MiB)
17:38:11.691 INFO SparkContext - Created broadcast 430 from broadcast at BamSink.java:76
17:38:11.694 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam.parts dst=null perm=null proto=rpc
17:38:11.694 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:11.694 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:11.694 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:11.695 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=mkdirs src=/user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam.parts/_temporary/0 dst=null perm=runner:supergroup:rwxr-xr-x proto=rpc
17:38:11.701 INFO SparkContext - Starting job: runJob at SparkHadoopWriter.scala:83
17:38:11.701 INFO DAGScheduler - Registering RDD 1028 (mapToPair at SparkUtils.java:161) as input to shuffle 43
17:38:11.701 INFO DAGScheduler - Got job 161 (runJob at SparkHadoopWriter.scala:83) with 1 output partitions
17:38:11.701 INFO DAGScheduler - Final stage: ResultStage 216 (runJob at SparkHadoopWriter.scala:83)
17:38:11.701 INFO DAGScheduler - Parents of final stage: List(ShuffleMapStage 215)
17:38:11.701 INFO DAGScheduler - Missing parents: List(ShuffleMapStage 215)
17:38:11.701 INFO DAGScheduler - Submitting ShuffleMapStage 215 (MapPartitionsRDD[1028] at mapToPair at SparkUtils.java:161), which has no missing parents
17:38:11.718 INFO MemoryStore - Block broadcast_431 stored as values in memory (estimated size 520.4 KiB, free 1917.1 MiB)
17:38:11.719 INFO MemoryStore - Block broadcast_431_piece0 stored as bytes in memory (estimated size 166.1 KiB, free 1916.9 MiB)
17:38:11.719 INFO BlockManagerInfo - Added broadcast_431_piece0 in memory on localhost:37953 (size: 166.1 KiB, free: 1919.5 MiB)
17:38:11.719 INFO SparkContext - Created broadcast 431 from broadcast at DAGScheduler.scala:1580
17:38:11.719 INFO DAGScheduler - Submitting 1 missing tasks from ShuffleMapStage 215 (MapPartitionsRDD[1028] at mapToPair at SparkUtils.java:161) (first 15 tasks are for partitions Vector(0))
17:38:11.719 INFO TaskSchedulerImpl - Adding task set 215.0 with 1 tasks resource profile 0
17:38:11.720 INFO TaskSetManager - Starting task 0.0 in stage 215.0 (TID 271) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7861 bytes)
17:38:11.720 INFO Executor - Running task 0.0 in stage 215.0 (TID 271)
17:38:11.754 INFO NewHadoopRDD - Input split: file:/home/runner/work/gatk/gatk/src/test/resources/org/broadinstitute/hellbender/tools/BQSR/HiSeq.1mb.1RG.2k_lines.bam:0+222075
17:38:11.755 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:11.756 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:11.763 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:11.771 INFO Executor - Finished task 0.0 in stage 215.0 (TID 271). 1148 bytes result sent to driver
17:38:11.772 INFO TaskSetManager - Finished task 0.0 in stage 215.0 (TID 271) in 52 ms on localhost (executor driver) (1/1)
17:38:11.772 INFO TaskSchedulerImpl - Removed TaskSet 215.0, whose tasks have all completed, from pool
17:38:11.772 INFO DAGScheduler - ShuffleMapStage 215 (mapToPair at SparkUtils.java:161) finished in 0.070 s
17:38:11.772 INFO DAGScheduler - looking for newly runnable stages
17:38:11.772 INFO DAGScheduler - running: HashSet()
17:38:11.772 INFO DAGScheduler - waiting: HashSet(ResultStage 216)
17:38:11.772 INFO DAGScheduler - failed: HashSet()
17:38:11.772 INFO DAGScheduler - Submitting ResultStage 216 (MapPartitionsRDD[1033] at mapToPair at BamSink.java:91), which has no missing parents
17:38:11.778 INFO MemoryStore - Block broadcast_432 stored as values in memory (estimated size 241.5 KiB, free 1916.7 MiB)
17:38:11.782 INFO BlockManagerInfo - Removed broadcast_423_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.5 MiB)
17:38:11.782 INFO MemoryStore - Block broadcast_432_piece0 stored as bytes in memory (estimated size 67.1 KiB, free 1916.9 MiB)
17:38:11.782 INFO BlockManagerInfo - Added broadcast_432_piece0 in memory on localhost:37953 (size: 67.1 KiB, free: 1919.4 MiB)
17:38:11.782 INFO SparkContext - Created broadcast 432 from broadcast at DAGScheduler.scala:1580
17:38:11.782 INFO BlockManagerInfo - Removed broadcast_425_piece0 on localhost:37953 in memory (size: 153.6 KiB, free: 1919.6 MiB)
17:38:11.782 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 216 (MapPartitionsRDD[1033] at mapToPair at BamSink.java:91) (first 15 tasks are for partitions Vector(0))
17:38:11.782 INFO TaskSchedulerImpl - Adding task set 216.0 with 1 tasks resource profile 0
17:38:11.783 INFO BlockManagerInfo - Removed broadcast_417_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.6 MiB)
17:38:11.783 INFO TaskSetManager - Starting task 0.0 in stage 216.0 (TID 272) (localhost, executor driver, partition 0, NODE_LOCAL, 7513 bytes)
17:38:11.783 INFO BlockManagerInfo - Removed broadcast_426_piece0 on localhost:37953 in memory (size: 3.8 KiB, free: 1919.6 MiB)
17:38:11.783 INFO Executor - Running task 0.0 in stage 216.0 (TID 272)
17:38:11.784 INFO BlockManagerInfo - Removed broadcast_422_piece0 on localhost:37953 in memory (size: 9.6 KiB, free: 1919.7 MiB)
17:38:11.785 INFO BlockManagerInfo - Removed broadcast_428_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.7 MiB)
17:38:11.788 INFO ShuffleBlockFetcherIterator - Getting 1 (343.8 KiB) non-empty blocks including 1 (343.8 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
17:38:11.788 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
17:38:11.799 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:11.799 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:11.799 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:11.800 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:11.800 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:11.800 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:11.801 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam.parts/_temporary/0/_temporary/attempt_202510271738118687552460505850839_1033_r_000000_0/part-r-00000 dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:11.801 WARN IntelDeflaterFactory - Intel Deflater not supported, using Java.util.zip.Deflater
17:38:11.802 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam.parts/_temporary/0/_temporary/attempt_202510271738118687552460505850839_1033_r_000000_0/.part-r-00000.sbi dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:11.803 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam.parts/_temporary/0/_temporary/attempt_202510271738118687552460505850839_1033_r_000000_0/.part-r-00000.bai dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:11.807 INFO StateChange - BLOCK* allocate blk_1073741871_1047, replicas=127.0.0.1:37927 for /user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam.parts/_temporary/0/_temporary/attempt_202510271738118687552460505850839_1033_r_000000_0/part-r-00000
17:38:11.808 INFO DataNode - Receiving BP-617366194-10.1.0.252-1761586664431:blk_1073741871_1047 src: /127.0.0.1:58024 dest: /127.0.0.1:37927
17:38:11.812 INFO clienttrace - src: /127.0.0.1:58024, dest: /127.0.0.1:37927, bytes: 233040, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1369393814_1, offset: 0, srvID: 379dd5e5-59c3-42cd-b9d3-67ca5e44560b, blockid: BP-617366194-10.1.0.252-1761586664431:blk_1073741871_1047, duration(ns): 3287860
17:38:11.812 INFO DataNode - PacketResponder: BP-617366194-10.1.0.252-1761586664431:blk_1073741871_1047, type=LAST_IN_PIPELINE terminating
17:38:11.813 INFO FSNamesystem - BLOCK* blk_1073741871_1047 is COMMITTED but not COMPLETE(numNodes= 0 < minimum = 1) in file /user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam.parts/_temporary/0/_temporary/attempt_202510271738118687552460505850839_1033_r_000000_0/part-r-00000
17:38:12.214 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam.parts/_temporary/0/_temporary/attempt_202510271738118687552460505850839_1033_r_000000_0/part-r-00000 is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:38:12.215 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam.parts/_temporary/0/_temporary/attempt_202510271738118687552460505850839_1033_r_000000_0/part-r-00000 dst=null perm=null proto=rpc
17:38:12.216 INFO StateChange - BLOCK* allocate blk_1073741872_1048, replicas=127.0.0.1:37927 for /user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam.parts/_temporary/0/_temporary/attempt_202510271738118687552460505850839_1033_r_000000_0/.part-r-00000.sbi
17:38:12.217 INFO DataNode - Receiving BP-617366194-10.1.0.252-1761586664431:blk_1073741872_1048 src: /127.0.0.1:58040 dest: /127.0.0.1:37927
17:38:12.218 INFO clienttrace - src: /127.0.0.1:58040, dest: /127.0.0.1:37927, bytes: 212, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1369393814_1, offset: 0, srvID: 379dd5e5-59c3-42cd-b9d3-67ca5e44560b, blockid: BP-617366194-10.1.0.252-1761586664431:blk_1073741872_1048, duration(ns): 599850
17:38:12.219 INFO DataNode - PacketResponder: BP-617366194-10.1.0.252-1761586664431:blk_1073741872_1048, type=LAST_IN_PIPELINE terminating
17:38:12.219 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam.parts/_temporary/0/_temporary/attempt_202510271738118687552460505850839_1033_r_000000_0/.part-r-00000.sbi is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:38:12.221 INFO StateChange - BLOCK* allocate blk_1073741873_1049, replicas=127.0.0.1:37927 for /user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam.parts/_temporary/0/_temporary/attempt_202510271738118687552460505850839_1033_r_000000_0/.part-r-00000.bai
17:38:12.221 INFO DataNode - Receiving BP-617366194-10.1.0.252-1761586664431:blk_1073741873_1049 src: /127.0.0.1:58054 dest: /127.0.0.1:37927
17:38:12.223 INFO clienttrace - src: /127.0.0.1:58054, dest: /127.0.0.1:37927, bytes: 5472, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1369393814_1, offset: 0, srvID: 379dd5e5-59c3-42cd-b9d3-67ca5e44560b, blockid: BP-617366194-10.1.0.252-1761586664431:blk_1073741873_1049, duration(ns): 517656
17:38:12.223 INFO DataNode - PacketResponder: BP-617366194-10.1.0.252-1761586664431:blk_1073741873_1049, type=LAST_IN_PIPELINE terminating
17:38:12.223 INFO FSNamesystem - BLOCK* blk_1073741873_1049 is COMMITTED but not COMPLETE(numNodes= 0 < minimum = 1) in file /user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam.parts/_temporary/0/_temporary/attempt_202510271738118687552460505850839_1033_r_000000_0/.part-r-00000.bai
17:38:12.624 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam.parts/_temporary/0/_temporary/attempt_202510271738118687552460505850839_1033_r_000000_0/.part-r-00000.bai is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:38:12.625 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam.parts/_temporary/0/_temporary/attempt_202510271738118687552460505850839_1033_r_000000_0 dst=null perm=null proto=rpc
17:38:12.626 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam.parts/_temporary/0/_temporary/attempt_202510271738118687552460505850839_1033_r_000000_0 dst=null perm=null proto=rpc
17:38:12.626 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam.parts/_temporary/0/task_202510271738118687552460505850839_1033_r_000000 dst=null perm=null proto=rpc
17:38:12.627 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=rename src=/user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam.parts/_temporary/0/_temporary/attempt_202510271738118687552460505850839_1033_r_000000_0 dst=/user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam.parts/_temporary/0/task_202510271738118687552460505850839_1033_r_000000 perm=runner:supergroup:rwxr-xr-x proto=rpc
17:38:12.627 INFO FileOutputCommitter - Saved output of task 'attempt_202510271738118687552460505850839_1033_r_000000_0' to hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam.parts/_temporary/0/task_202510271738118687552460505850839_1033_r_000000
17:38:12.627 INFO SparkHadoopMapRedUtil - attempt_202510271738118687552460505850839_1033_r_000000_0: Committed. Elapsed time: 2 ms.
17:38:12.628 INFO Executor - Finished task 0.0 in stage 216.0 (TID 272). 1858 bytes result sent to driver
17:38:12.628 INFO TaskSetManager - Finished task 0.0 in stage 216.0 (TID 272) in 845 ms on localhost (executor driver) (1/1)
17:38:12.628 INFO TaskSchedulerImpl - Removed TaskSet 216.0, whose tasks have all completed, from pool
17:38:12.629 INFO DAGScheduler - ResultStage 216 (runJob at SparkHadoopWriter.scala:83) finished in 0.857 s
17:38:12.629 INFO DAGScheduler - Job 161 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:12.629 INFO TaskSchedulerImpl - Killing all running tasks in stage 216: Stage finished
17:38:12.629 INFO DAGScheduler - Job 161 finished: runJob at SparkHadoopWriter.scala:83, took 0.928191 s
17:38:12.629 INFO SparkHadoopWriter - Start to commit write Job job_202510271738118687552460505850839_1033.
17:38:12.630 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=listStatus src=/user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam.parts/_temporary/0 dst=null perm=null proto=rpc
17:38:12.630 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam.parts dst=null perm=null proto=rpc
17:38:12.631 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=listStatus src=/user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam.parts/_temporary/0/task_202510271738118687552460505850839_1033_r_000000 dst=null perm=null proto=rpc
17:38:12.631 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam.parts/.part-r-00000.bai dst=null perm=null proto=rpc
17:38:12.632 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=rename src=/user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam.parts/_temporary/0/task_202510271738118687552460505850839_1033_r_000000/.part-r-00000.bai dst=/user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam.parts/.part-r-00000.bai perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:12.632 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam.parts/.part-r-00000.sbi dst=null perm=null proto=rpc
17:38:12.633 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=rename src=/user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam.parts/_temporary/0/task_202510271738118687552460505850839_1033_r_000000/.part-r-00000.sbi dst=/user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam.parts/.part-r-00000.sbi perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:12.633 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam.parts/part-r-00000 dst=null perm=null proto=rpc
17:38:12.634 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=rename src=/user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam.parts/_temporary/0/task_202510271738118687552460505850839_1033_r_000000/part-r-00000 dst=/user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam.parts/part-r-00000 perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:12.635 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=delete src=/user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam.parts/_temporary dst=null perm=null proto=rpc
17:38:12.635 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam.parts/_SUCCESS dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:12.636 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam.parts/_SUCCESS is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:38:12.637 INFO audit - allowed=false ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=delete src=/user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam.parts/.spark-staging-1033 dst=null perm=null proto=rpc
17:38:12.637 INFO SparkHadoopWriter - Write Job job_202510271738118687552460505850839_1033 committed. Elapsed time: 8 ms.
17:38:12.638 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam.parts/header dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:12.638 WARN IntelDeflaterFactory - Intel Deflater not supported, using Java.util.zip.Deflater
17:38:12.639 INFO StateChange - BLOCK* allocate blk_1073741874_1050, replicas=127.0.0.1:37927 for /user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam.parts/header
17:38:12.640 INFO DataNode - Receiving BP-617366194-10.1.0.252-1761586664431:blk_1073741874_1050 src: /127.0.0.1:58066 dest: /127.0.0.1:37927
17:38:12.642 INFO clienttrace - src: /127.0.0.1:58066, dest: /127.0.0.1:37927, bytes: 5901, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1369393814_1, offset: 0, srvID: 379dd5e5-59c3-42cd-b9d3-67ca5e44560b, blockid: BP-617366194-10.1.0.252-1761586664431:blk_1073741874_1050, duration(ns): 485687
17:38:12.642 INFO DataNode - PacketResponder: BP-617366194-10.1.0.252-1761586664431:blk_1073741874_1050, type=LAST_IN_PIPELINE terminating
17:38:12.643 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam.parts/header is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:38:12.643 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam.parts/terminator dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:12.644 INFO StateChange - BLOCK* allocate blk_1073741875_1051, replicas=127.0.0.1:37927 for /user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam.parts/terminator
17:38:12.645 INFO DataNode - Receiving BP-617366194-10.1.0.252-1761586664431:blk_1073741875_1051 src: /127.0.0.1:58082 dest: /127.0.0.1:37927
17:38:12.646 INFO clienttrace - src: /127.0.0.1:58082, dest: /127.0.0.1:37927, bytes: 28, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1369393814_1, offset: 0, srvID: 379dd5e5-59c3-42cd-b9d3-67ca5e44560b, blockid: BP-617366194-10.1.0.252-1761586664431:blk_1073741875_1051, duration(ns): 409685
17:38:12.646 INFO DataNode - PacketResponder: BP-617366194-10.1.0.252-1761586664431:blk_1073741875_1051, type=LAST_IN_PIPELINE terminating
17:38:12.646 INFO FSNamesystem - BLOCK* blk_1073741875_1051 is COMMITTED but not COMPLETE(numNodes= 0 < minimum = 1) in file /user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam.parts/terminator
17:38:13.048 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam.parts/terminator is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:38:13.049 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=listStatus src=/user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam.parts dst=null perm=null proto=rpc
17:38:13.050 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam.parts/output dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:13.050 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam.parts/output is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:38:13.051 INFO HadoopFileSystemWrapper - Concatenating 3 parts to hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam
17:38:13.051 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=concat src=[/user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam.parts/header, /user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam.parts/part-r-00000, /user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam.parts/terminator] dst=/user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam.parts/output perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:13.052 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam dst=null perm=null proto=rpc
17:38:13.052 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=delete src=/user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam dst=null perm=null proto=rpc
17:38:13.053 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=rename src=/user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam.parts/output dst=/user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:13.053 INFO HadoopFileSystemWrapper - Concatenating to hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam done
17:38:13.053 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam dst=null perm=null proto=rpc
17:38:13.053 INFO IndexFileMerger - Merging .sbi files in temp directory hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam.parts/ to hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam.sbi
17:38:13.054 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=listStatus src=/user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam.parts dst=null perm=null proto=rpc
17:38:13.055 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam.sbi dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:13.056 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam.parts/.part-r-00000.sbi dst=null perm=null proto=rpc
17:38:13.056 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam.parts/.part-r-00000.sbi dst=null perm=null proto=rpc
17:38:13.058 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=delete src=/user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam.parts/.part-r-00000.sbi dst=null perm=null proto=rpc
17:38:13.059 INFO StateChange - BLOCK* allocate blk_1073741876_1052, replicas=127.0.0.1:37927 for /user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam.sbi
17:38:13.060 INFO DataNode - Receiving BP-617366194-10.1.0.252-1761586664431:blk_1073741876_1052 src: /127.0.0.1:58094 dest: /127.0.0.1:37927
17:38:13.061 INFO clienttrace - src: /127.0.0.1:58094, dest: /127.0.0.1:37927, bytes: 212, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1369393814_1, offset: 0, srvID: 379dd5e5-59c3-42cd-b9d3-67ca5e44560b, blockid: BP-617366194-10.1.0.252-1761586664431:blk_1073741876_1052, duration(ns): 488111
17:38:13.061 INFO DataNode - PacketResponder: BP-617366194-10.1.0.252-1761586664431:blk_1073741876_1052, type=LAST_IN_PIPELINE terminating
17:38:13.062 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam.sbi is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:38:13.062 INFO IndexFileMerger - Done merging .sbi files
17:38:13.062 INFO IndexFileMerger - Merging .bai files in temp directory hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam.parts/ to hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam.bai
17:38:13.062 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=listStatus src=/user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam.parts dst=null perm=null proto=rpc
17:38:13.063 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam.bai dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:13.064 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam.parts/.part-r-00000.bai dst=null perm=null proto=rpc
17:38:13.065 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam.parts/.part-r-00000.bai dst=null perm=null proto=rpc
17:38:13.066 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=delete src=/user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam.parts/.part-r-00000.bai dst=null perm=null proto=rpc
17:38:13.067 INFO StateChange - BLOCK* allocate blk_1073741877_1053, replicas=127.0.0.1:37927 for /user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam.bai
17:38:13.068 INFO DataNode - Receiving BP-617366194-10.1.0.252-1761586664431:blk_1073741877_1053 src: /127.0.0.1:58108 dest: /127.0.0.1:37927
17:38:13.069 INFO clienttrace - src: /127.0.0.1:58108, dest: /127.0.0.1:37927, bytes: 5472, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1369393814_1, offset: 0, srvID: 379dd5e5-59c3-42cd-b9d3-67ca5e44560b, blockid: BP-617366194-10.1.0.252-1761586664431:blk_1073741877_1053, duration(ns): 828217
17:38:13.069 INFO DataNode - PacketResponder: BP-617366194-10.1.0.252-1761586664431:blk_1073741877_1053, type=LAST_IN_PIPELINE terminating
17:38:13.070 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam.bai is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:38:13.070 INFO IndexFileMerger - Done merging .bai files
17:38:13.071 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=delete src=/user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam.parts dst=null perm=null proto=rpc
17:38:13.079 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam.bai dst=null perm=null proto=rpc
17:38:13.087 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam.sbi dst=null perm=null proto=rpc
17:38:13.087 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam.sbi dst=null perm=null proto=rpc
17:38:13.088 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam.sbi dst=null perm=null proto=rpc
17:38:13.089 WARN DFSUtil - Unexpected value for data transfer bytes=216 duration=0
17:38:13.089 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam dst=null perm=null proto=rpc
17:38:13.089 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam dst=null perm=null proto=rpc
17:38:13.090 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam dst=null perm=null proto=rpc
17:38:13.091 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam dst=null perm=null proto=rpc
17:38:13.091 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam.bai dst=null perm=null proto=rpc
17:38:13.092 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam.bai dst=null perm=null proto=rpc
17:38:13.092 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam.bai dst=null perm=null proto=rpc
17:38:13.093 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:13.094 WARN DFSUtil - Unexpected value for data transfer bytes=5949 duration=0
17:38:13.096 WARN DFSUtil - Unexpected value for data transfer bytes=5516 duration=0
17:38:13.096 WARN DFSUtil - Unexpected value for data transfer bytes=5516 duration=0
17:38:13.096 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:13.097 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam.sbi dst=null perm=null proto=rpc
17:38:13.097 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam.sbi dst=null perm=null proto=rpc
17:38:13.098 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam.sbi dst=null perm=null proto=rpc
17:38:13.098 WARN DFSUtil - Unexpected value for data transfer bytes=216 duration=0
17:38:13.099 INFO MemoryStore - Block broadcast_433 stored as values in memory (estimated size 320.0 B, free 1918.4 MiB)
17:38:13.099 INFO MemoryStore - Block broadcast_433_piece0 stored as bytes in memory (estimated size 233.0 B, free 1918.4 MiB)
17:38:13.099 INFO BlockManagerInfo - Added broadcast_433_piece0 in memory on localhost:37953 (size: 233.0 B, free: 1919.7 MiB)
17:38:13.099 INFO SparkContext - Created broadcast 433 from broadcast at BamSource.java:104
17:38:13.100 INFO MemoryStore - Block broadcast_434 stored as values in memory (estimated size 297.9 KiB, free 1918.1 MiB)
17:38:13.106 INFO MemoryStore - Block broadcast_434_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1918.0 MiB)
17:38:13.106 INFO BlockManagerInfo - Added broadcast_434_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.7 MiB)
17:38:13.106 INFO SparkContext - Created broadcast 434 from newAPIHadoopFile at PathSplitSource.java:96
17:38:13.115 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam dst=null perm=null proto=rpc
17:38:13.115 INFO FileInputFormat - Total input files to process : 1
17:38:13.115 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam dst=null perm=null proto=rpc
17:38:13.129 INFO SparkContext - Starting job: collect at ReadsSparkSinkUnitTest.java:182
17:38:13.129 INFO DAGScheduler - Got job 162 (collect at ReadsSparkSinkUnitTest.java:182) with 1 output partitions
17:38:13.129 INFO DAGScheduler - Final stage: ResultStage 217 (collect at ReadsSparkSinkUnitTest.java:182)
17:38:13.129 INFO DAGScheduler - Parents of final stage: List()
17:38:13.129 INFO DAGScheduler - Missing parents: List()
17:38:13.129 INFO DAGScheduler - Submitting ResultStage 217 (MapPartitionsRDD[1039] at filter at ReadsSparkSource.java:96), which has no missing parents
17:38:13.134 INFO MemoryStore - Block broadcast_435 stored as values in memory (estimated size 148.2 KiB, free 1917.9 MiB)
17:38:13.135 INFO MemoryStore - Block broadcast_435_piece0 stored as bytes in memory (estimated size 54.6 KiB, free 1917.8 MiB)
17:38:13.135 INFO BlockManagerInfo - Added broadcast_435_piece0 in memory on localhost:37953 (size: 54.6 KiB, free: 1919.6 MiB)
17:38:13.135 INFO SparkContext - Created broadcast 435 from broadcast at DAGScheduler.scala:1580
17:38:13.135 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 217 (MapPartitionsRDD[1039] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:38:13.135 INFO TaskSchedulerImpl - Adding task set 217.0 with 1 tasks resource profile 0
17:38:13.136 INFO TaskSetManager - Starting task 0.0 in stage 217.0 (TID 273) (localhost, executor driver, partition 0, ANY, 7852 bytes)
17:38:13.136 INFO Executor - Running task 0.0 in stage 217.0 (TID 273)
17:38:13.147 INFO NewHadoopRDD - Input split: hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam:0+238969
17:38:13.148 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam dst=null perm=null proto=rpc
17:38:13.149 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam dst=null perm=null proto=rpc
17:38:13.149 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam.bai dst=null perm=null proto=rpc
17:38:13.150 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam.bai dst=null perm=null proto=rpc
17:38:13.150 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam.bai dst=null perm=null proto=rpc
17:38:13.151 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:13.152 WARN DFSUtil - Unexpected value for data transfer bytes=5949 duration=0
17:38:13.154 WARN DFSUtil - Unexpected value for data transfer bytes=5516 duration=0
17:38:13.154 WARN DFSUtil - Unexpected value for data transfer bytes=5516 duration=0
17:38:13.156 WARN DFSUtil - Unexpected value for data transfer bytes=32 duration=0
17:38:13.158 INFO Executor - Finished task 0.0 in stage 217.0 (TID 273). 651483 bytes result sent to driver
17:38:13.160 INFO TaskSetManager - Finished task 0.0 in stage 217.0 (TID 273) in 24 ms on localhost (executor driver) (1/1)
17:38:13.160 INFO TaskSchedulerImpl - Removed TaskSet 217.0, whose tasks have all completed, from pool
17:38:13.160 INFO DAGScheduler - ResultStage 217 (collect at ReadsSparkSinkUnitTest.java:182) finished in 0.031 s
17:38:13.160 INFO DAGScheduler - Job 162 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:13.160 INFO TaskSchedulerImpl - Killing all running tasks in stage 217: Stage finished
17:38:13.160 INFO DAGScheduler - Job 162 finished: collect at ReadsSparkSinkUnitTest.java:182, took 0.031516 s
17:38:13.170 INFO SparkContext - Starting job: count at ReadsSparkSinkUnitTest.java:185
17:38:13.170 INFO DAGScheduler - Got job 163 (count at ReadsSparkSinkUnitTest.java:185) with 1 output partitions
17:38:13.170 INFO DAGScheduler - Final stage: ResultStage 218 (count at ReadsSparkSinkUnitTest.java:185)
17:38:13.170 INFO DAGScheduler - Parents of final stage: List()
17:38:13.170 INFO DAGScheduler - Missing parents: List()
17:38:13.170 INFO DAGScheduler - Submitting ResultStage 218 (MapPartitionsRDD[1021] at filter at ReadsSparkSource.java:96), which has no missing parents
17:38:13.185 INFO MemoryStore - Block broadcast_436 stored as values in memory (estimated size 426.1 KiB, free 1917.4 MiB)
17:38:13.186 INFO MemoryStore - Block broadcast_436_piece0 stored as bytes in memory (estimated size 153.6 KiB, free 1917.2 MiB)
17:38:13.186 INFO BlockManagerInfo - Added broadcast_436_piece0 in memory on localhost:37953 (size: 153.6 KiB, free: 1919.5 MiB)
17:38:13.186 INFO SparkContext - Created broadcast 436 from broadcast at DAGScheduler.scala:1580
17:38:13.186 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 218 (MapPartitionsRDD[1021] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:38:13.186 INFO TaskSchedulerImpl - Adding task set 218.0 with 1 tasks resource profile 0
17:38:13.187 INFO TaskSetManager - Starting task 0.0 in stage 218.0 (TID 274) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7872 bytes)
17:38:13.187 INFO Executor - Running task 0.0 in stage 218.0 (TID 274)
17:38:13.217 INFO NewHadoopRDD - Input split: file:/home/runner/work/gatk/gatk/src/test/resources/org/broadinstitute/hellbender/tools/BQSR/HiSeq.1mb.1RG.2k_lines.bam:0+222075
17:38:13.218 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:13.219 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:13.226 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:13.229 INFO Executor - Finished task 0.0 in stage 218.0 (TID 274). 989 bytes result sent to driver
17:38:13.229 INFO TaskSetManager - Finished task 0.0 in stage 218.0 (TID 274) in 42 ms on localhost (executor driver) (1/1)
17:38:13.229 INFO TaskSchedulerImpl - Removed TaskSet 218.0, whose tasks have all completed, from pool
17:38:13.230 INFO DAGScheduler - ResultStage 218 (count at ReadsSparkSinkUnitTest.java:185) finished in 0.060 s
17:38:13.230 INFO DAGScheduler - Job 163 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:13.230 INFO TaskSchedulerImpl - Killing all running tasks in stage 218: Stage finished
17:38:13.230 INFO DAGScheduler - Job 163 finished: count at ReadsSparkSinkUnitTest.java:185, took 0.060021 s
17:38:13.233 INFO SparkContext - Starting job: count at ReadsSparkSinkUnitTest.java:185
17:38:13.233 INFO DAGScheduler - Got job 164 (count at ReadsSparkSinkUnitTest.java:185) with 1 output partitions
17:38:13.233 INFO DAGScheduler - Final stage: ResultStage 219 (count at ReadsSparkSinkUnitTest.java:185)
17:38:13.233 INFO DAGScheduler - Parents of final stage: List()
17:38:13.233 INFO DAGScheduler - Missing parents: List()
17:38:13.233 INFO DAGScheduler - Submitting ResultStage 219 (MapPartitionsRDD[1039] at filter at ReadsSparkSource.java:96), which has no missing parents
17:38:13.239 INFO MemoryStore - Block broadcast_437 stored as values in memory (estimated size 148.1 KiB, free 1917.1 MiB)
17:38:13.239 INFO MemoryStore - Block broadcast_437_piece0 stored as bytes in memory (estimated size 54.6 KiB, free 1917.1 MiB)
17:38:13.239 INFO BlockManagerInfo - Added broadcast_437_piece0 in memory on localhost:37953 (size: 54.6 KiB, free: 1919.4 MiB)
17:38:13.239 INFO SparkContext - Created broadcast 437 from broadcast at DAGScheduler.scala:1580
17:38:13.240 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 219 (MapPartitionsRDD[1039] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:38:13.240 INFO TaskSchedulerImpl - Adding task set 219.0 with 1 tasks resource profile 0
17:38:13.240 INFO TaskSetManager - Starting task 0.0 in stage 219.0 (TID 275) (localhost, executor driver, partition 0, ANY, 7852 bytes)
17:38:13.240 INFO Executor - Running task 0.0 in stage 219.0 (TID 275)
17:38:13.251 INFO NewHadoopRDD - Input split: hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam:0+238969
17:38:13.251 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam dst=null perm=null proto=rpc
17:38:13.252 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam dst=null perm=null proto=rpc
17:38:13.253 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam.bai dst=null perm=null proto=rpc
17:38:13.254 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam.bai dst=null perm=null proto=rpc
17:38:13.254 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_4f7b3cd9-44e3-4327-b61b-5b0c406bd441.bam.bai dst=null perm=null proto=rpc
17:38:13.255 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:13.256 WARN DFSUtil - Unexpected value for data transfer bytes=5949 duration=0
17:38:13.257 WARN DFSUtil - Unexpected value for data transfer bytes=5516 duration=0
17:38:13.258 WARN DFSUtil - Unexpected value for data transfer bytes=5516 duration=0
17:38:13.259 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:13.260 WARN DFSUtil - Unexpected value for data transfer bytes=32 duration=0
17:38:13.265 INFO Executor - Finished task 0.0 in stage 219.0 (TID 275). 1075 bytes result sent to driver
17:38:13.265 INFO BlockManagerInfo - Removed broadcast_429_piece0 on localhost:37953 in memory (size: 9.6 KiB, free: 1919.4 MiB)
17:38:13.265 INFO TaskSetManager - Finished task 0.0 in stage 219.0 (TID 275) in 25 ms on localhost (executor driver) (1/1)
17:38:13.265 INFO TaskSchedulerImpl - Removed TaskSet 219.0, whose tasks have all completed, from pool
17:38:13.265 INFO DAGScheduler - ResultStage 219 (count at ReadsSparkSinkUnitTest.java:185) finished in 0.032 s
17:38:13.265 INFO DAGScheduler - Job 164 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:13.265 INFO TaskSchedulerImpl - Killing all running tasks in stage 219: Stage finished
17:38:13.265 INFO DAGScheduler - Job 164 finished: count at ReadsSparkSinkUnitTest.java:185, took 0.032706 s
17:38:13.267 INFO BlockManagerInfo - Removed broadcast_435_piece0 on localhost:37953 in memory (size: 54.6 KiB, free: 1919.5 MiB)
17:38:13.267 INFO BlockManagerInfo - Removed broadcast_436_piece0 on localhost:37953 in memory (size: 153.6 KiB, free: 1919.6 MiB)
17:38:13.268 INFO BlockManagerInfo - Removed broadcast_431_piece0 on localhost:37953 in memory (size: 166.1 KiB, free: 1919.8 MiB)
17:38:13.268 INFO BlockManagerInfo - Removed broadcast_430_piece0 on localhost:37953 in memory (size: 9.6 KiB, free: 1919.8 MiB)
17:38:13.271 INFO BlockManagerInfo - Removed broadcast_432_piece0 on localhost:37953 in memory (size: 67.1 KiB, free: 1919.8 MiB)
17:38:13.274 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam dst=null perm=null proto=rpc
17:38:13.275 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:13.276 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:38:13.276 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam dst=null perm=null proto=rpc
17:38:13.277 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:13.279 INFO MemoryStore - Block broadcast_438 stored as values in memory (estimated size 297.9 KiB, free 1918.8 MiB)
17:38:13.284 INFO MemoryStore - Block broadcast_438_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1918.8 MiB)
17:38:13.285 INFO BlockManagerInfo - Added broadcast_438_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.8 MiB)
17:38:13.285 INFO SparkContext - Created broadcast 438 from newAPIHadoopFile at PathSplitSource.java:96
17:38:13.304 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:13.305 INFO MemoryStore - Block broadcast_439 stored as values in memory (estimated size 297.9 KiB, free 1918.5 MiB)
17:38:13.311 INFO MemoryStore - Block broadcast_439_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1918.4 MiB)
17:38:13.311 INFO BlockManagerInfo - Added broadcast_439_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.8 MiB)
17:38:13.311 INFO SparkContext - Created broadcast 439 from newAPIHadoopFile at PathSplitSource.java:96
17:38:13.330 INFO FileInputFormat - Total input files to process : 1
17:38:13.332 INFO MemoryStore - Block broadcast_440 stored as values in memory (estimated size 160.7 KiB, free 1918.3 MiB)
17:38:13.332 INFO MemoryStore - Block broadcast_440_piece0 stored as bytes in memory (estimated size 9.6 KiB, free 1918.3 MiB)
17:38:13.332 INFO BlockManagerInfo - Added broadcast_440_piece0 in memory on localhost:37953 (size: 9.6 KiB, free: 1919.7 MiB)
17:38:13.333 INFO SparkContext - Created broadcast 440 from broadcast at ReadsSparkSink.java:133
17:38:13.334 INFO MemoryStore - Block broadcast_441 stored as values in memory (estimated size 163.2 KiB, free 1918.1 MiB)
17:38:13.334 INFO MemoryStore - Block broadcast_441_piece0 stored as bytes in memory (estimated size 9.6 KiB, free 1918.1 MiB)
17:38:13.335 INFO BlockManagerInfo - Added broadcast_441_piece0 in memory on localhost:37953 (size: 9.6 KiB, free: 1919.7 MiB)
17:38:13.335 INFO SparkContext - Created broadcast 441 from broadcast at BamSink.java:76
17:38:13.337 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam.parts dst=null perm=null proto=rpc
17:38:13.337 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:13.337 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:13.337 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:13.338 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=mkdirs src=/user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam.parts/_temporary/0 dst=null perm=runner:supergroup:rwxr-xr-x proto=rpc
17:38:13.344 INFO SparkContext - Starting job: runJob at SparkHadoopWriter.scala:83
17:38:13.344 INFO DAGScheduler - Registering RDD 1053 (mapToPair at SparkUtils.java:161) as input to shuffle 44
17:38:13.344 INFO DAGScheduler - Got job 165 (runJob at SparkHadoopWriter.scala:83) with 1 output partitions
17:38:13.344 INFO DAGScheduler - Final stage: ResultStage 221 (runJob at SparkHadoopWriter.scala:83)
17:38:13.344 INFO DAGScheduler - Parents of final stage: List(ShuffleMapStage 220)
17:38:13.344 INFO DAGScheduler - Missing parents: List(ShuffleMapStage 220)
17:38:13.345 INFO DAGScheduler - Submitting ShuffleMapStage 220 (MapPartitionsRDD[1053] at mapToPair at SparkUtils.java:161), which has no missing parents
17:38:13.360 INFO MemoryStore - Block broadcast_442 stored as values in memory (estimated size 520.4 KiB, free 1917.6 MiB)
17:38:13.361 INFO MemoryStore - Block broadcast_442_piece0 stored as bytes in memory (estimated size 166.1 KiB, free 1917.4 MiB)
17:38:13.361 INFO BlockManagerInfo - Added broadcast_442_piece0 in memory on localhost:37953 (size: 166.1 KiB, free: 1919.6 MiB)
17:38:13.362 INFO SparkContext - Created broadcast 442 from broadcast at DAGScheduler.scala:1580
17:38:13.362 INFO DAGScheduler - Submitting 1 missing tasks from ShuffleMapStage 220 (MapPartitionsRDD[1053] at mapToPair at SparkUtils.java:161) (first 15 tasks are for partitions Vector(0))
17:38:13.362 INFO TaskSchedulerImpl - Adding task set 220.0 with 1 tasks resource profile 0
17:38:13.362 INFO TaskSetManager - Starting task 0.0 in stage 220.0 (TID 276) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7861 bytes)
17:38:13.363 INFO Executor - Running task 0.0 in stage 220.0 (TID 276)
17:38:13.394 INFO NewHadoopRDD - Input split: file:/home/runner/work/gatk/gatk/src/test/resources/org/broadinstitute/hellbender/tools/BQSR/HiSeq.1mb.1RG.2k_lines.bam:0+222075
17:38:13.395 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:13.395 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:13.402 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:13.411 INFO Executor - Finished task 0.0 in stage 220.0 (TID 276). 1148 bytes result sent to driver
17:38:13.411 INFO TaskSetManager - Finished task 0.0 in stage 220.0 (TID 276) in 49 ms on localhost (executor driver) (1/1)
17:38:13.411 INFO TaskSchedulerImpl - Removed TaskSet 220.0, whose tasks have all completed, from pool
17:38:13.411 INFO DAGScheduler - ShuffleMapStage 220 (mapToPair at SparkUtils.java:161) finished in 0.066 s
17:38:13.411 INFO DAGScheduler - looking for newly runnable stages
17:38:13.411 INFO DAGScheduler - running: HashSet()
17:38:13.411 INFO DAGScheduler - waiting: HashSet(ResultStage 221)
17:38:13.411 INFO DAGScheduler - failed: HashSet()
17:38:13.411 INFO DAGScheduler - Submitting ResultStage 221 (MapPartitionsRDD[1058] at mapToPair at BamSink.java:91), which has no missing parents
17:38:13.417 INFO MemoryStore - Block broadcast_443 stored as values in memory (estimated size 241.5 KiB, free 1917.2 MiB)
17:38:13.418 INFO MemoryStore - Block broadcast_443_piece0 stored as bytes in memory (estimated size 67.1 KiB, free 1917.1 MiB)
17:38:13.418 INFO BlockManagerInfo - Added broadcast_443_piece0 in memory on localhost:37953 (size: 67.1 KiB, free: 1919.5 MiB)
17:38:13.418 INFO SparkContext - Created broadcast 443 from broadcast at DAGScheduler.scala:1580
17:38:13.418 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 221 (MapPartitionsRDD[1058] at mapToPair at BamSink.java:91) (first 15 tasks are for partitions Vector(0))
17:38:13.418 INFO TaskSchedulerImpl - Adding task set 221.0 with 1 tasks resource profile 0
17:38:13.419 INFO TaskSetManager - Starting task 0.0 in stage 221.0 (TID 277) (localhost, executor driver, partition 0, NODE_LOCAL, 7513 bytes)
17:38:13.419 INFO Executor - Running task 0.0 in stage 221.0 (TID 277)
17:38:13.424 INFO ShuffleBlockFetcherIterator - Getting 1 (343.8 KiB) non-empty blocks including 1 (343.8 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
17:38:13.424 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
17:38:13.435 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:13.435 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:13.435 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:13.435 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:13.435 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:13.435 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:13.436 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam.parts/_temporary/0/_temporary/attempt_202510271738137526693437293419930_1058_r_000000_0/part-r-00000 dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:13.437 WARN IntelDeflaterFactory - Intel Deflater not supported, using Java.util.zip.Deflater
17:38:13.438 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam.parts/_temporary/0/_temporary/attempt_202510271738137526693437293419930_1058_r_000000_0/.part-r-00000.sbi dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:13.439 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam.parts/_temporary/0/_temporary/attempt_202510271738137526693437293419930_1058_r_000000_0/.part-r-00000.bai dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:13.442 INFO StateChange - BLOCK* allocate blk_1073741878_1054, replicas=127.0.0.1:37927 for /user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam.parts/_temporary/0/_temporary/attempt_202510271738137526693437293419930_1058_r_000000_0/part-r-00000
17:38:13.443 INFO DataNode - Receiving BP-617366194-10.1.0.252-1761586664431:blk_1073741878_1054 src: /127.0.0.1:58136 dest: /127.0.0.1:37927
17:38:13.447 INFO clienttrace - src: /127.0.0.1:58136, dest: /127.0.0.1:37927, bytes: 233040, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1369393814_1, offset: 0, srvID: 379dd5e5-59c3-42cd-b9d3-67ca5e44560b, blockid: BP-617366194-10.1.0.252-1761586664431:blk_1073741878_1054, duration(ns): 3444729
17:38:13.447 INFO DataNode - PacketResponder: BP-617366194-10.1.0.252-1761586664431:blk_1073741878_1054, type=LAST_IN_PIPELINE terminating
17:38:13.448 INFO FSNamesystem - BLOCK* blk_1073741878_1054 is COMMITTED but not COMPLETE(numNodes= 0 < minimum = 1) in file /user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam.parts/_temporary/0/_temporary/attempt_202510271738137526693437293419930_1058_r_000000_0/part-r-00000
17:38:13.849 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam.parts/_temporary/0/_temporary/attempt_202510271738137526693437293419930_1058_r_000000_0/part-r-00000 is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:38:13.850 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam.parts/_temporary/0/_temporary/attempt_202510271738137526693437293419930_1058_r_000000_0/part-r-00000 dst=null perm=null proto=rpc
17:38:13.851 INFO StateChange - BLOCK* allocate blk_1073741879_1055, replicas=127.0.0.1:37927 for /user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam.parts/_temporary/0/_temporary/attempt_202510271738137526693437293419930_1058_r_000000_0/.part-r-00000.sbi
17:38:13.852 INFO DataNode - Receiving BP-617366194-10.1.0.252-1761586664431:blk_1073741879_1055 src: /127.0.0.1:58140 dest: /127.0.0.1:37927
17:38:13.854 INFO clienttrace - src: /127.0.0.1:58140, dest: /127.0.0.1:37927, bytes: 13492, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1369393814_1, offset: 0, srvID: 379dd5e5-59c3-42cd-b9d3-67ca5e44560b, blockid: BP-617366194-10.1.0.252-1761586664431:blk_1073741879_1055, duration(ns): 581041
17:38:13.854 INFO DataNode - PacketResponder: BP-617366194-10.1.0.252-1761586664431:blk_1073741879_1055, type=LAST_IN_PIPELINE terminating
17:38:13.854 INFO FSNamesystem - BLOCK* blk_1073741879_1055 is COMMITTED but not COMPLETE(numNodes= 0 < minimum = 1) in file /user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam.parts/_temporary/0/_temporary/attempt_202510271738137526693437293419930_1058_r_000000_0/.part-r-00000.sbi
17:38:14.255 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam.parts/_temporary/0/_temporary/attempt_202510271738137526693437293419930_1058_r_000000_0/.part-r-00000.sbi is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:38:14.257 INFO StateChange - BLOCK* allocate blk_1073741880_1056, replicas=127.0.0.1:37927 for /user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam.parts/_temporary/0/_temporary/attempt_202510271738137526693437293419930_1058_r_000000_0/.part-r-00000.bai
17:38:14.258 INFO DataNode - Receiving BP-617366194-10.1.0.252-1761586664431:blk_1073741880_1056 src: /127.0.0.1:58150 dest: /127.0.0.1:37927
17:38:14.259 INFO clienttrace - src: /127.0.0.1:58150, dest: /127.0.0.1:37927, bytes: 5472, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1369393814_1, offset: 0, srvID: 379dd5e5-59c3-42cd-b9d3-67ca5e44560b, blockid: BP-617366194-10.1.0.252-1761586664431:blk_1073741880_1056, duration(ns): 551993
17:38:14.259 INFO DataNode - PacketResponder: BP-617366194-10.1.0.252-1761586664431:blk_1073741880_1056, type=LAST_IN_PIPELINE terminating
17:38:14.260 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam.parts/_temporary/0/_temporary/attempt_202510271738137526693437293419930_1058_r_000000_0/.part-r-00000.bai is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:38:14.261 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam.parts/_temporary/0/_temporary/attempt_202510271738137526693437293419930_1058_r_000000_0 dst=null perm=null proto=rpc
17:38:14.261 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam.parts/_temporary/0/_temporary/attempt_202510271738137526693437293419930_1058_r_000000_0 dst=null perm=null proto=rpc
17:38:14.262 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam.parts/_temporary/0/task_202510271738137526693437293419930_1058_r_000000 dst=null perm=null proto=rpc
17:38:14.262 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=rename src=/user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam.parts/_temporary/0/_temporary/attempt_202510271738137526693437293419930_1058_r_000000_0 dst=/user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam.parts/_temporary/0/task_202510271738137526693437293419930_1058_r_000000 perm=runner:supergroup:rwxr-xr-x proto=rpc
17:38:14.263 INFO FileOutputCommitter - Saved output of task 'attempt_202510271738137526693437293419930_1058_r_000000_0' to hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam.parts/_temporary/0/task_202510271738137526693437293419930_1058_r_000000
17:38:14.263 INFO SparkHadoopMapRedUtil - attempt_202510271738137526693437293419930_1058_r_000000_0: Committed. Elapsed time: 1 ms.
17:38:14.263 INFO Executor - Finished task 0.0 in stage 221.0 (TID 277). 1858 bytes result sent to driver
17:38:14.264 INFO TaskSetManager - Finished task 0.0 in stage 221.0 (TID 277) in 845 ms on localhost (executor driver) (1/1)
17:38:14.264 INFO TaskSchedulerImpl - Removed TaskSet 221.0, whose tasks have all completed, from pool
17:38:14.264 INFO DAGScheduler - ResultStage 221 (runJob at SparkHadoopWriter.scala:83) finished in 0.852 s
17:38:14.264 INFO DAGScheduler - Job 165 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:14.264 INFO TaskSchedulerImpl - Killing all running tasks in stage 221: Stage finished
17:38:14.264 INFO DAGScheduler - Job 165 finished: runJob at SparkHadoopWriter.scala:83, took 0.920154 s
17:38:14.264 INFO SparkHadoopWriter - Start to commit write Job job_202510271738137526693437293419930_1058.
17:38:14.265 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=listStatus src=/user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam.parts/_temporary/0 dst=null perm=null proto=rpc
17:38:14.266 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam.parts dst=null perm=null proto=rpc
17:38:14.266 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=listStatus src=/user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam.parts/_temporary/0/task_202510271738137526693437293419930_1058_r_000000 dst=null perm=null proto=rpc
17:38:14.267 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam.parts/.part-r-00000.bai dst=null perm=null proto=rpc
17:38:14.267 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=rename src=/user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam.parts/_temporary/0/task_202510271738137526693437293419930_1058_r_000000/.part-r-00000.bai dst=/user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam.parts/.part-r-00000.bai perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:14.268 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam.parts/.part-r-00000.sbi dst=null perm=null proto=rpc
17:38:14.268 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=rename src=/user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam.parts/_temporary/0/task_202510271738137526693437293419930_1058_r_000000/.part-r-00000.sbi dst=/user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam.parts/.part-r-00000.sbi perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:14.269 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam.parts/part-r-00000 dst=null perm=null proto=rpc
17:38:14.269 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=rename src=/user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam.parts/_temporary/0/task_202510271738137526693437293419930_1058_r_000000/part-r-00000 dst=/user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam.parts/part-r-00000 perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:14.270 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=delete src=/user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam.parts/_temporary dst=null perm=null proto=rpc
17:38:14.270 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam.parts/_SUCCESS dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:14.271 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam.parts/_SUCCESS is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:38:14.272 INFO audit - allowed=false ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=delete src=/user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam.parts/.spark-staging-1058 dst=null perm=null proto=rpc
17:38:14.272 INFO SparkHadoopWriter - Write Job job_202510271738137526693437293419930_1058 committed. Elapsed time: 7 ms.
17:38:14.273 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam.parts/header dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:14.273 WARN IntelDeflaterFactory - Intel Deflater not supported, using Java.util.zip.Deflater
17:38:14.274 INFO StateChange - BLOCK* allocate blk_1073741881_1057, replicas=127.0.0.1:37927 for /user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam.parts/header
17:38:14.275 INFO DataNode - Receiving BP-617366194-10.1.0.252-1761586664431:blk_1073741881_1057 src: /127.0.0.1:58156 dest: /127.0.0.1:37927
17:38:14.276 INFO clienttrace - src: /127.0.0.1:58156, dest: /127.0.0.1:37927, bytes: 5901, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1369393814_1, offset: 0, srvID: 379dd5e5-59c3-42cd-b9d3-67ca5e44560b, blockid: BP-617366194-10.1.0.252-1761586664431:blk_1073741881_1057, duration(ns): 469206
17:38:14.276 INFO DataNode - PacketResponder: BP-617366194-10.1.0.252-1761586664431:blk_1073741881_1057, type=LAST_IN_PIPELINE terminating
17:38:14.277 INFO FSNamesystem - BLOCK* blk_1073741881_1057 is COMMITTED but not COMPLETE(numNodes= 0 < minimum = 1) in file /user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam.parts/header
17:38:14.678 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam.parts/header is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:38:14.679 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam.parts/terminator dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:14.680 INFO StateChange - BLOCK* allocate blk_1073741882_1058, replicas=127.0.0.1:37927 for /user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam.parts/terminator
17:38:14.681 INFO DataNode - Receiving BP-617366194-10.1.0.252-1761586664431:blk_1073741882_1058 src: /127.0.0.1:58160 dest: /127.0.0.1:37927
17:38:14.682 INFO clienttrace - src: /127.0.0.1:58160, dest: /127.0.0.1:37927, bytes: 28, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1369393814_1, offset: 0, srvID: 379dd5e5-59c3-42cd-b9d3-67ca5e44560b, blockid: BP-617366194-10.1.0.252-1761586664431:blk_1073741882_1058, duration(ns): 575666
17:38:14.682 INFO DataNode - PacketResponder: BP-617366194-10.1.0.252-1761586664431:blk_1073741882_1058, type=LAST_IN_PIPELINE terminating
17:38:14.683 INFO FSNamesystem - BLOCK* blk_1073741882_1058 is COMMITTED but not COMPLETE(numNodes= 0 < minimum = 1) in file /user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam.parts/terminator
17:38:15.084 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam.parts/terminator is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:38:15.085 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=listStatus src=/user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam.parts dst=null perm=null proto=rpc
17:38:15.086 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam.parts/output dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:15.087 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam.parts/output is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:38:15.087 INFO HadoopFileSystemWrapper - Concatenating 3 parts to hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam
17:38:15.087 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=concat src=[/user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam.parts/header, /user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam.parts/part-r-00000, /user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam.parts/terminator] dst=/user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam.parts/output perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:15.088 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam dst=null perm=null proto=rpc
17:38:15.089 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=delete src=/user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam dst=null perm=null proto=rpc
17:38:15.089 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=rename src=/user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam.parts/output dst=/user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:15.089 INFO HadoopFileSystemWrapper - Concatenating to hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam done
17:38:15.090 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam dst=null perm=null proto=rpc
17:38:15.090 INFO IndexFileMerger - Merging .sbi files in temp directory hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam.parts/ to hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam.sbi
17:38:15.090 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=listStatus src=/user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam.parts dst=null perm=null proto=rpc
17:38:15.091 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam.sbi dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:15.092 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam.parts/.part-r-00000.sbi dst=null perm=null proto=rpc
17:38:15.093 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam.parts/.part-r-00000.sbi dst=null perm=null proto=rpc
17:38:15.094 WARN DFSUtil - Unexpected value for data transfer bytes=13600 duration=0
17:38:15.094 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=delete src=/user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam.parts/.part-r-00000.sbi dst=null perm=null proto=rpc
17:38:15.095 INFO StateChange - BLOCK* allocate blk_1073741883_1059, replicas=127.0.0.1:37927 for /user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam.sbi
17:38:15.096 INFO DataNode - Receiving BP-617366194-10.1.0.252-1761586664431:blk_1073741883_1059 src: /127.0.0.1:58176 dest: /127.0.0.1:37927
17:38:15.097 INFO clienttrace - src: /127.0.0.1:58176, dest: /127.0.0.1:37927, bytes: 13492, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1369393814_1, offset: 0, srvID: 379dd5e5-59c3-42cd-b9d3-67ca5e44560b, blockid: BP-617366194-10.1.0.252-1761586664431:blk_1073741883_1059, duration(ns): 585874
17:38:15.097 INFO DataNode - PacketResponder: BP-617366194-10.1.0.252-1761586664431:blk_1073741883_1059, type=LAST_IN_PIPELINE terminating
17:38:15.098 INFO FSNamesystem - BLOCK* blk_1073741883_1059 is COMMITTED but not COMPLETE(numNodes= 0 < minimum = 1) in file /user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam.sbi
17:38:15.502 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam.sbi is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:38:15.512 INFO IndexFileMerger - Done merging .sbi files
17:38:15.512 INFO IndexFileMerger - Merging .bai files in temp directory hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam.parts/ to hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam.bai
17:38:15.513 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=listStatus src=/user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam.parts dst=null perm=null proto=rpc
17:38:15.514 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam.bai dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:15.514 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam.parts/.part-r-00000.bai dst=null perm=null proto=rpc
17:38:15.515 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam.parts/.part-r-00000.bai dst=null perm=null proto=rpc
17:38:15.516 WARN DFSUtil - Unexpected value for data transfer bytes=5516 duration=0
17:38:15.517 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=delete src=/user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam.parts/.part-r-00000.bai dst=null perm=null proto=rpc
17:38:15.518 INFO StateChange - BLOCK* allocate blk_1073741884_1060, replicas=127.0.0.1:37927 for /user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam.bai
17:38:15.519 INFO DataNode - Receiving BP-617366194-10.1.0.252-1761586664431:blk_1073741884_1060 src: /127.0.0.1:58184 dest: /127.0.0.1:37927
17:38:15.520 INFO clienttrace - src: /127.0.0.1:58184, dest: /127.0.0.1:37927, bytes: 5472, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1369393814_1, offset: 0, srvID: 379dd5e5-59c3-42cd-b9d3-67ca5e44560b, blockid: BP-617366194-10.1.0.252-1761586664431:blk_1073741884_1060, duration(ns): 553985
17:38:15.520 INFO DataNode - PacketResponder: BP-617366194-10.1.0.252-1761586664431:blk_1073741884_1060, type=LAST_IN_PIPELINE terminating
17:38:15.521 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam.bai is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:38:15.521 INFO IndexFileMerger - Done merging .bai files
17:38:15.522 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=delete src=/user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam.parts dst=null perm=null proto=rpc
17:38:15.530 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam.bai dst=null perm=null proto=rpc
17:38:15.537 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam.sbi dst=null perm=null proto=rpc
17:38:15.537 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam.sbi dst=null perm=null proto=rpc
17:38:15.538 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam.sbi dst=null perm=null proto=rpc
17:38:15.539 WARN DFSUtil - Unexpected value for data transfer bytes=13600 duration=0
17:38:15.539 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam dst=null perm=null proto=rpc
17:38:15.540 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam dst=null perm=null proto=rpc
17:38:15.540 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam dst=null perm=null proto=rpc
17:38:15.541 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam dst=null perm=null proto=rpc
17:38:15.541 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam.bai dst=null perm=null proto=rpc
17:38:15.542 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam.bai dst=null perm=null proto=rpc
17:38:15.542 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam.bai dst=null perm=null proto=rpc
17:38:15.543 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:15.544 WARN DFSUtil - Unexpected value for data transfer bytes=5949 duration=0
17:38:15.545 WARN DFSUtil - Unexpected value for data transfer bytes=5516 duration=0
17:38:15.546 WARN DFSUtil - Unexpected value for data transfer bytes=5516 duration=0
17:38:15.546 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:15.546 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam.sbi dst=null perm=null proto=rpc
17:38:15.547 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam.sbi dst=null perm=null proto=rpc
17:38:15.547 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam.sbi dst=null perm=null proto=rpc
17:38:15.548 WARN DFSUtil - Unexpected value for data transfer bytes=13600 duration=0
17:38:15.549 INFO MemoryStore - Block broadcast_444 stored as values in memory (estimated size 13.3 KiB, free 1917.1 MiB)
17:38:15.552 INFO MemoryStore - Block broadcast_444_piece0 stored as bytes in memory (estimated size 8.3 KiB, free 1917.1 MiB)
17:38:15.552 INFO BlockManagerInfo - Added broadcast_444_piece0 in memory on localhost:37953 (size: 8.3 KiB, free: 1919.5 MiB)
17:38:15.552 INFO BlockManagerInfo - Removed broadcast_439_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.5 MiB)
17:38:15.552 INFO SparkContext - Created broadcast 444 from broadcast at BamSource.java:104
17:38:15.553 INFO BlockManagerInfo - Removed broadcast_442_piece0 on localhost:37953 in memory (size: 166.1 KiB, free: 1919.7 MiB)
17:38:15.553 INFO BlockManagerInfo - Removed broadcast_434_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.8 MiB)
17:38:15.553 INFO MemoryStore - Block broadcast_445 stored as values in memory (estimated size 297.9 KiB, free 1918.2 MiB)
17:38:15.554 INFO BlockManagerInfo - Removed broadcast_437_piece0 on localhost:37953 in memory (size: 54.6 KiB, free: 1919.8 MiB)
17:38:15.554 INFO BlockManagerInfo - Removed broadcast_433_piece0 on localhost:37953 in memory (size: 233.0 B, free: 1919.8 MiB)
17:38:15.555 INFO BlockManagerInfo - Removed broadcast_443_piece0 on localhost:37953 in memory (size: 67.1 KiB, free: 1919.9 MiB)
17:38:15.556 INFO BlockManagerInfo - Removed broadcast_440_piece0 on localhost:37953 in memory (size: 9.6 KiB, free: 1919.9 MiB)
17:38:15.556 INFO BlockManagerInfo - Removed broadcast_441_piece0 on localhost:37953 in memory (size: 9.6 KiB, free: 1919.9 MiB)
17:38:15.556 INFO BlockManagerInfo - Removed broadcast_427_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.9 MiB)
17:38:15.559 INFO MemoryStore - Block broadcast_445_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1919.3 MiB)
17:38:15.559 INFO BlockManagerInfo - Added broadcast_445_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.9 MiB)
17:38:15.559 INFO SparkContext - Created broadcast 445 from newAPIHadoopFile at PathSplitSource.java:96
17:38:15.568 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam dst=null perm=null proto=rpc
17:38:15.568 INFO FileInputFormat - Total input files to process : 1
17:38:15.568 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam dst=null perm=null proto=rpc
17:38:15.582 INFO SparkContext - Starting job: collect at ReadsSparkSinkUnitTest.java:182
17:38:15.582 INFO DAGScheduler - Got job 166 (collect at ReadsSparkSinkUnitTest.java:182) with 1 output partitions
17:38:15.582 INFO DAGScheduler - Final stage: ResultStage 222 (collect at ReadsSparkSinkUnitTest.java:182)
17:38:15.582 INFO DAGScheduler - Parents of final stage: List()
17:38:15.582 INFO DAGScheduler - Missing parents: List()
17:38:15.582 INFO DAGScheduler - Submitting ResultStage 222 (MapPartitionsRDD[1064] at filter at ReadsSparkSource.java:96), which has no missing parents
17:38:15.588 INFO MemoryStore - Block broadcast_446 stored as values in memory (estimated size 148.2 KiB, free 1919.2 MiB)
17:38:15.588 INFO MemoryStore - Block broadcast_446_piece0 stored as bytes in memory (estimated size 54.6 KiB, free 1919.1 MiB)
17:38:15.589 INFO BlockManagerInfo - Added broadcast_446_piece0 in memory on localhost:37953 (size: 54.6 KiB, free: 1919.8 MiB)
17:38:15.589 INFO SparkContext - Created broadcast 446 from broadcast at DAGScheduler.scala:1580
17:38:15.589 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 222 (MapPartitionsRDD[1064] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:38:15.589 INFO TaskSchedulerImpl - Adding task set 222.0 with 1 tasks resource profile 0
17:38:15.590 INFO TaskSetManager - Starting task 0.0 in stage 222.0 (TID 278) (localhost, executor driver, partition 0, ANY, 7852 bytes)
17:38:15.590 INFO Executor - Running task 0.0 in stage 222.0 (TID 278)
17:38:15.599 INFO FsDatasetAsyncDiskService - Scheduling blk_1073741872_1048 replica FinalizedReplica, blk_1073741872_1048, FINALIZED
getNumBytes() = 212
getBytesOnDisk() = 212
getVisibleLength()= 212
getVolume() = /tmp/minicluster_storage14473397491642437130/data/data2
getBlockURI() = file:/tmp/minicluster_storage14473397491642437130/data/data2/current/BP-617366194-10.1.0.252-1761586664431/current/finalized/subdir0/subdir0/blk_1073741872 for deletion
17:38:15.599 INFO FsDatasetAsyncDiskService - Scheduling blk_1073741873_1049 replica FinalizedReplica, blk_1073741873_1049, FINALIZED
getNumBytes() = 5472
getBytesOnDisk() = 5472
getVisibleLength()= 5472
getVolume() = /tmp/minicluster_storage14473397491642437130/data/data1
getBlockURI() = file:/tmp/minicluster_storage14473397491642437130/data/data1/current/BP-617366194-10.1.0.252-1761586664431/current/finalized/subdir0/subdir0/blk_1073741873 for deletion
17:38:15.599 INFO FsDatasetAsyncDiskService - Deleted BP-617366194-10.1.0.252-1761586664431 blk_1073741872_1048 URI file:/tmp/minicluster_storage14473397491642437130/data/data2/current/BP-617366194-10.1.0.252-1761586664431/current/finalized/subdir0/subdir0/blk_1073741872
17:38:15.600 INFO FsDatasetAsyncDiskService - Deleted BP-617366194-10.1.0.252-1761586664431 blk_1073741873_1049 URI file:/tmp/minicluster_storage14473397491642437130/data/data1/current/BP-617366194-10.1.0.252-1761586664431/current/finalized/subdir0/subdir0/blk_1073741873
17:38:15.602 INFO NewHadoopRDD - Input split: hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam:0+238969
17:38:15.602 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam dst=null perm=null proto=rpc
17:38:15.603 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam dst=null perm=null proto=rpc
17:38:15.603 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam.bai dst=null perm=null proto=rpc
17:38:15.604 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam.bai dst=null perm=null proto=rpc
17:38:15.605 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam.bai dst=null perm=null proto=rpc
17:38:15.606 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:15.606 WARN DFSUtil - Unexpected value for data transfer bytes=5949 duration=0
17:38:15.608 WARN DFSUtil - Unexpected value for data transfer bytes=5516 duration=0
17:38:15.609 WARN DFSUtil - Unexpected value for data transfer bytes=5516 duration=0
17:38:15.610 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:15.613 INFO Executor - Finished task 0.0 in stage 222.0 (TID 278). 651483 bytes result sent to driver
17:38:15.614 INFO TaskSetManager - Finished task 0.0 in stage 222.0 (TID 278) in 25 ms on localhost (executor driver) (1/1)
17:38:15.614 INFO TaskSchedulerImpl - Removed TaskSet 222.0, whose tasks have all completed, from pool
17:38:15.614 INFO DAGScheduler - ResultStage 222 (collect at ReadsSparkSinkUnitTest.java:182) finished in 0.031 s
17:38:15.614 INFO DAGScheduler - Job 166 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:15.614 INFO TaskSchedulerImpl - Killing all running tasks in stage 222: Stage finished
17:38:15.614 INFO DAGScheduler - Job 166 finished: collect at ReadsSparkSinkUnitTest.java:182, took 0.032624 s
17:38:15.624 INFO SparkContext - Starting job: count at ReadsSparkSinkUnitTest.java:185
17:38:15.624 INFO DAGScheduler - Got job 167 (count at ReadsSparkSinkUnitTest.java:185) with 1 output partitions
17:38:15.624 INFO DAGScheduler - Final stage: ResultStage 223 (count at ReadsSparkSinkUnitTest.java:185)
17:38:15.624 INFO DAGScheduler - Parents of final stage: List()
17:38:15.624 INFO DAGScheduler - Missing parents: List()
17:38:15.625 INFO DAGScheduler - Submitting ResultStage 223 (MapPartitionsRDD[1046] at filter at ReadsSparkSource.java:96), which has no missing parents
17:38:15.640 INFO MemoryStore - Block broadcast_447 stored as values in memory (estimated size 426.1 KiB, free 1918.7 MiB)
17:38:15.641 INFO MemoryStore - Block broadcast_447_piece0 stored as bytes in memory (estimated size 153.6 KiB, free 1918.5 MiB)
17:38:15.641 INFO BlockManagerInfo - Added broadcast_447_piece0 in memory on localhost:37953 (size: 153.6 KiB, free: 1919.7 MiB)
17:38:15.641 INFO SparkContext - Created broadcast 447 from broadcast at DAGScheduler.scala:1580
17:38:15.641 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 223 (MapPartitionsRDD[1046] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:38:15.641 INFO TaskSchedulerImpl - Adding task set 223.0 with 1 tasks resource profile 0
17:38:15.642 INFO TaskSetManager - Starting task 0.0 in stage 223.0 (TID 279) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7872 bytes)
17:38:15.642 INFO Executor - Running task 0.0 in stage 223.0 (TID 279)
17:38:15.672 INFO NewHadoopRDD - Input split: file:/home/runner/work/gatk/gatk/src/test/resources/org/broadinstitute/hellbender/tools/BQSR/HiSeq.1mb.1RG.2k_lines.bam:0+222075
17:38:15.673 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:15.674 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:15.681 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:15.684 INFO Executor - Finished task 0.0 in stage 223.0 (TID 279). 989 bytes result sent to driver
17:38:15.685 INFO TaskSetManager - Finished task 0.0 in stage 223.0 (TID 279) in 43 ms on localhost (executor driver) (1/1)
17:38:15.685 INFO TaskSchedulerImpl - Removed TaskSet 223.0, whose tasks have all completed, from pool
17:38:15.685 INFO DAGScheduler - ResultStage 223 (count at ReadsSparkSinkUnitTest.java:185) finished in 0.060 s
17:38:15.685 INFO DAGScheduler - Job 167 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:15.685 INFO TaskSchedulerImpl - Killing all running tasks in stage 223: Stage finished
17:38:15.685 INFO DAGScheduler - Job 167 finished: count at ReadsSparkSinkUnitTest.java:185, took 0.061109 s
17:38:15.688 INFO SparkContext - Starting job: count at ReadsSparkSinkUnitTest.java:185
17:38:15.688 INFO DAGScheduler - Got job 168 (count at ReadsSparkSinkUnitTest.java:185) with 1 output partitions
17:38:15.689 INFO DAGScheduler - Final stage: ResultStage 224 (count at ReadsSparkSinkUnitTest.java:185)
17:38:15.689 INFO DAGScheduler - Parents of final stage: List()
17:38:15.689 INFO DAGScheduler - Missing parents: List()
17:38:15.689 INFO DAGScheduler - Submitting ResultStage 224 (MapPartitionsRDD[1064] at filter at ReadsSparkSource.java:96), which has no missing parents
17:38:15.694 INFO MemoryStore - Block broadcast_448 stored as values in memory (estimated size 148.1 KiB, free 1918.4 MiB)
17:38:15.695 INFO MemoryStore - Block broadcast_448_piece0 stored as bytes in memory (estimated size 54.6 KiB, free 1918.3 MiB)
17:38:15.695 INFO BlockManagerInfo - Added broadcast_448_piece0 in memory on localhost:37953 (size: 54.6 KiB, free: 1919.6 MiB)
17:38:15.695 INFO SparkContext - Created broadcast 448 from broadcast at DAGScheduler.scala:1580
17:38:15.695 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 224 (MapPartitionsRDD[1064] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:38:15.695 INFO TaskSchedulerImpl - Adding task set 224.0 with 1 tasks resource profile 0
17:38:15.696 INFO TaskSetManager - Starting task 0.0 in stage 224.0 (TID 280) (localhost, executor driver, partition 0, ANY, 7852 bytes)
17:38:15.696 INFO Executor - Running task 0.0 in stage 224.0 (TID 280)
17:38:15.707 INFO NewHadoopRDD - Input split: hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam:0+238969
17:38:15.707 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam dst=null perm=null proto=rpc
17:38:15.708 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam dst=null perm=null proto=rpc
17:38:15.709 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam.bai dst=null perm=null proto=rpc
17:38:15.709 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam.bai dst=null perm=null proto=rpc
17:38:15.710 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_9340a345-aca6-4a07-bdc4-5f3cb5d4d1b8.bam.bai dst=null perm=null proto=rpc
17:38:15.711 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:15.711 WARN DFSUtil - Unexpected value for data transfer bytes=5949 duration=0
17:38:15.713 WARN DFSUtil - Unexpected value for data transfer bytes=5516 duration=0
17:38:15.715 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:15.715 WARN DFSUtil - Unexpected value for data transfer bytes=32 duration=0
17:38:15.717 INFO Executor - Finished task 0.0 in stage 224.0 (TID 280). 989 bytes result sent to driver
17:38:15.717 INFO TaskSetManager - Finished task 0.0 in stage 224.0 (TID 280) in 21 ms on localhost (executor driver) (1/1)
17:38:15.717 INFO TaskSchedulerImpl - Removed TaskSet 224.0, whose tasks have all completed, from pool
17:38:15.717 INFO DAGScheduler - ResultStage 224 (count at ReadsSparkSinkUnitTest.java:185) finished in 0.028 s
17:38:15.717 INFO DAGScheduler - Job 168 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:15.717 INFO TaskSchedulerImpl - Killing all running tasks in stage 224: Stage finished
17:38:15.717 INFO DAGScheduler - Job 168 finished: count at ReadsSparkSinkUnitTest.java:185, took 0.029105 s
17:38:15.725 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam dst=null perm=null proto=rpc
17:38:15.726 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:15.727 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:38:15.727 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam dst=null perm=null proto=rpc
17:38:15.728 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:15.730 INFO MemoryStore - Block broadcast_449 stored as values in memory (estimated size 297.9 KiB, free 1918.0 MiB)
17:38:15.736 INFO MemoryStore - Block broadcast_449_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1918.0 MiB)
17:38:15.736 INFO BlockManagerInfo - Added broadcast_449_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.6 MiB)
17:38:15.736 INFO SparkContext - Created broadcast 449 from newAPIHadoopFile at PathSplitSource.java:96
17:38:15.755 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:15.757 INFO MemoryStore - Block broadcast_450 stored as values in memory (estimated size 297.9 KiB, free 1917.7 MiB)
17:38:15.762 INFO BlockManagerInfo - Removed broadcast_445_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.6 MiB)
17:38:15.763 INFO BlockManagerInfo - Removed broadcast_444_piece0 on localhost:37953 in memory (size: 8.3 KiB, free: 1919.6 MiB)
17:38:15.763 INFO BlockManagerInfo - Removed broadcast_438_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.7 MiB)
17:38:15.764 INFO BlockManagerInfo - Removed broadcast_446_piece0 on localhost:37953 in memory (size: 54.6 KiB, free: 1919.7 MiB)
17:38:15.764 INFO BlockManagerInfo - Removed broadcast_447_piece0 on localhost:37953 in memory (size: 153.6 KiB, free: 1919.9 MiB)
17:38:15.764 INFO BlockManagerInfo - Removed broadcast_448_piece0 on localhost:37953 in memory (size: 54.6 KiB, free: 1920.0 MiB)
17:38:15.766 INFO MemoryStore - Block broadcast_450_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1919.3 MiB)
17:38:15.766 INFO BlockManagerInfo - Added broadcast_450_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.9 MiB)
17:38:15.766 INFO SparkContext - Created broadcast 450 from newAPIHadoopFile at PathSplitSource.java:96
17:38:15.785 INFO FileInputFormat - Total input files to process : 1
17:38:15.787 INFO MemoryStore - Block broadcast_451 stored as values in memory (estimated size 160.7 KiB, free 1919.2 MiB)
17:38:15.788 INFO MemoryStore - Block broadcast_451_piece0 stored as bytes in memory (estimated size 9.6 KiB, free 1919.2 MiB)
17:38:15.788 INFO BlockManagerInfo - Added broadcast_451_piece0 in memory on localhost:37953 (size: 9.6 KiB, free: 1919.9 MiB)
17:38:15.788 INFO SparkContext - Created broadcast 451 from broadcast at ReadsSparkSink.java:133
17:38:15.789 WARN HtsjdkReadsRddStorage - Unrecognized write option: DISABLE
17:38:15.789 INFO MemoryStore - Block broadcast_452 stored as values in memory (estimated size 163.2 KiB, free 1919.0 MiB)
17:38:15.790 INFO MemoryStore - Block broadcast_452_piece0 stored as bytes in memory (estimated size 9.6 KiB, free 1919.0 MiB)
17:38:15.790 INFO BlockManagerInfo - Added broadcast_452_piece0 in memory on localhost:37953 (size: 9.6 KiB, free: 1919.9 MiB)
17:38:15.790 INFO SparkContext - Created broadcast 452 from broadcast at BamSink.java:76
17:38:15.793 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam.parts dst=null perm=null proto=rpc
17:38:15.793 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:15.793 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:15.793 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:15.794 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=mkdirs src=/user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam.parts/_temporary/0 dst=null perm=runner:supergroup:rwxr-xr-x proto=rpc
17:38:15.799 INFO SparkContext - Starting job: runJob at SparkHadoopWriter.scala:83
17:38:15.800 INFO DAGScheduler - Registering RDD 1078 (mapToPair at SparkUtils.java:161) as input to shuffle 45
17:38:15.800 INFO DAGScheduler - Got job 169 (runJob at SparkHadoopWriter.scala:83) with 1 output partitions
17:38:15.800 INFO DAGScheduler - Final stage: ResultStage 226 (runJob at SparkHadoopWriter.scala:83)
17:38:15.800 INFO DAGScheduler - Parents of final stage: List(ShuffleMapStage 225)
17:38:15.800 INFO DAGScheduler - Missing parents: List(ShuffleMapStage 225)
17:38:15.800 INFO DAGScheduler - Submitting ShuffleMapStage 225 (MapPartitionsRDD[1078] at mapToPair at SparkUtils.java:161), which has no missing parents
17:38:15.816 INFO MemoryStore - Block broadcast_453 stored as values in memory (estimated size 520.4 KiB, free 1918.5 MiB)
17:38:15.817 INFO MemoryStore - Block broadcast_453_piece0 stored as bytes in memory (estimated size 166.1 KiB, free 1918.3 MiB)
17:38:15.817 INFO BlockManagerInfo - Added broadcast_453_piece0 in memory on localhost:37953 (size: 166.1 KiB, free: 1919.7 MiB)
17:38:15.817 INFO SparkContext - Created broadcast 453 from broadcast at DAGScheduler.scala:1580
17:38:15.817 INFO DAGScheduler - Submitting 1 missing tasks from ShuffleMapStage 225 (MapPartitionsRDD[1078] at mapToPair at SparkUtils.java:161) (first 15 tasks are for partitions Vector(0))
17:38:15.818 INFO TaskSchedulerImpl - Adding task set 225.0 with 1 tasks resource profile 0
17:38:15.818 INFO TaskSetManager - Starting task 0.0 in stage 225.0 (TID 281) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7861 bytes)
17:38:15.818 INFO Executor - Running task 0.0 in stage 225.0 (TID 281)
17:38:15.851 INFO NewHadoopRDD - Input split: file:/home/runner/work/gatk/gatk/src/test/resources/org/broadinstitute/hellbender/tools/BQSR/HiSeq.1mb.1RG.2k_lines.bam:0+222075
17:38:15.852 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:15.853 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:15.860 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:15.869 INFO Executor - Finished task 0.0 in stage 225.0 (TID 281). 1148 bytes result sent to driver
17:38:15.869 INFO TaskSetManager - Finished task 0.0 in stage 225.0 (TID 281) in 51 ms on localhost (executor driver) (1/1)
17:38:15.869 INFO TaskSchedulerImpl - Removed TaskSet 225.0, whose tasks have all completed, from pool
17:38:15.869 INFO DAGScheduler - ShuffleMapStage 225 (mapToPair at SparkUtils.java:161) finished in 0.069 s
17:38:15.869 INFO DAGScheduler - looking for newly runnable stages
17:38:15.869 INFO DAGScheduler - running: HashSet()
17:38:15.869 INFO DAGScheduler - waiting: HashSet(ResultStage 226)
17:38:15.870 INFO DAGScheduler - failed: HashSet()
17:38:15.870 INFO DAGScheduler - Submitting ResultStage 226 (MapPartitionsRDD[1083] at mapToPair at BamSink.java:91), which has no missing parents
17:38:15.876 INFO MemoryStore - Block broadcast_454 stored as values in memory (estimated size 241.5 KiB, free 1918.1 MiB)
17:38:15.876 INFO MemoryStore - Block broadcast_454_piece0 stored as bytes in memory (estimated size 67.1 KiB, free 1918.0 MiB)
17:38:15.877 INFO BlockManagerInfo - Added broadcast_454_piece0 in memory on localhost:37953 (size: 67.1 KiB, free: 1919.7 MiB)
17:38:15.877 INFO SparkContext - Created broadcast 454 from broadcast at DAGScheduler.scala:1580
17:38:15.877 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 226 (MapPartitionsRDD[1083] at mapToPair at BamSink.java:91) (first 15 tasks are for partitions Vector(0))
17:38:15.877 INFO TaskSchedulerImpl - Adding task set 226.0 with 1 tasks resource profile 0
17:38:15.878 INFO TaskSetManager - Starting task 0.0 in stage 226.0 (TID 282) (localhost, executor driver, partition 0, NODE_LOCAL, 7513 bytes)
17:38:15.878 INFO Executor - Running task 0.0 in stage 226.0 (TID 282)
17:38:15.883 INFO ShuffleBlockFetcherIterator - Getting 1 (343.8 KiB) non-empty blocks including 1 (343.8 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
17:38:15.883 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
17:38:15.894 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:15.894 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:15.894 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:15.894 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:15.894 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:15.894 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:15.895 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam.parts/_temporary/0/_temporary/attempt_202510271738159192766228709143977_1083_r_000000_0/part-r-00000 dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:15.901 WARN IntelDeflaterFactory - Intel Deflater not supported, using Java.util.zip.Deflater
17:38:15.902 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam.parts/_temporary/0/_temporary/attempt_202510271738159192766228709143977_1083_r_000000_0/.part-r-00000.bai dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:15.906 INFO StateChange - BLOCK* allocate blk_1073741885_1061, replicas=127.0.0.1:37927 for /user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam.parts/_temporary/0/_temporary/attempt_202510271738159192766228709143977_1083_r_000000_0/part-r-00000
17:38:15.907 INFO DataNode - Receiving BP-617366194-10.1.0.252-1761586664431:blk_1073741885_1061 src: /127.0.0.1:58212 dest: /127.0.0.1:37927
17:38:15.911 INFO clienttrace - src: /127.0.0.1:58212, dest: /127.0.0.1:37927, bytes: 233040, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1369393814_1, offset: 0, srvID: 379dd5e5-59c3-42cd-b9d3-67ca5e44560b, blockid: BP-617366194-10.1.0.252-1761586664431:blk_1073741885_1061, duration(ns): 3143632
17:38:15.911 INFO DataNode - PacketResponder: BP-617366194-10.1.0.252-1761586664431:blk_1073741885_1061, type=LAST_IN_PIPELINE terminating
17:38:15.912 INFO FSNamesystem - BLOCK* blk_1073741885_1061 is COMMITTED but not COMPLETE(numNodes= 0 < minimum = 1) in file /user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam.parts/_temporary/0/_temporary/attempt_202510271738159192766228709143977_1083_r_000000_0/part-r-00000
17:38:16.312 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam.parts/_temporary/0/_temporary/attempt_202510271738159192766228709143977_1083_r_000000_0/part-r-00000 is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:38:16.313 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam.parts/_temporary/0/_temporary/attempt_202510271738159192766228709143977_1083_r_000000_0/part-r-00000 dst=null perm=null proto=rpc
17:38:16.315 INFO StateChange - BLOCK* allocate blk_1073741886_1062, replicas=127.0.0.1:37927 for /user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam.parts/_temporary/0/_temporary/attempt_202510271738159192766228709143977_1083_r_000000_0/.part-r-00000.bai
17:38:16.316 INFO DataNode - Receiving BP-617366194-10.1.0.252-1761586664431:blk_1073741886_1062 src: /127.0.0.1:58224 dest: /127.0.0.1:37927
17:38:16.317 INFO clienttrace - src: /127.0.0.1:58224, dest: /127.0.0.1:37927, bytes: 5472, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1369393814_1, offset: 0, srvID: 379dd5e5-59c3-42cd-b9d3-67ca5e44560b, blockid: BP-617366194-10.1.0.252-1761586664431:blk_1073741886_1062, duration(ns): 573906
17:38:16.317 INFO DataNode - PacketResponder: BP-617366194-10.1.0.252-1761586664431:blk_1073741886_1062, type=LAST_IN_PIPELINE terminating
17:38:16.318 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam.parts/_temporary/0/_temporary/attempt_202510271738159192766228709143977_1083_r_000000_0/.part-r-00000.bai is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:38:16.319 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam.parts/_temporary/0/_temporary/attempt_202510271738159192766228709143977_1083_r_000000_0 dst=null perm=null proto=rpc
17:38:16.319 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam.parts/_temporary/0/_temporary/attempt_202510271738159192766228709143977_1083_r_000000_0 dst=null perm=null proto=rpc
17:38:16.320 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam.parts/_temporary/0/task_202510271738159192766228709143977_1083_r_000000 dst=null perm=null proto=rpc
17:38:16.320 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=rename src=/user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam.parts/_temporary/0/_temporary/attempt_202510271738159192766228709143977_1083_r_000000_0 dst=/user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam.parts/_temporary/0/task_202510271738159192766228709143977_1083_r_000000 perm=runner:supergroup:rwxr-xr-x proto=rpc
17:38:16.320 INFO FileOutputCommitter - Saved output of task 'attempt_202510271738159192766228709143977_1083_r_000000_0' to hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam.parts/_temporary/0/task_202510271738159192766228709143977_1083_r_000000
17:38:16.321 INFO SparkHadoopMapRedUtil - attempt_202510271738159192766228709143977_1083_r_000000_0: Committed. Elapsed time: 1 ms.
17:38:16.321 INFO Executor - Finished task 0.0 in stage 226.0 (TID 282). 1858 bytes result sent to driver
17:38:16.321 INFO TaskSetManager - Finished task 0.0 in stage 226.0 (TID 282) in 444 ms on localhost (executor driver) (1/1)
17:38:16.321 INFO TaskSchedulerImpl - Removed TaskSet 226.0, whose tasks have all completed, from pool
17:38:16.322 INFO DAGScheduler - ResultStage 226 (runJob at SparkHadoopWriter.scala:83) finished in 0.452 s
17:38:16.322 INFO DAGScheduler - Job 169 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:16.322 INFO TaskSchedulerImpl - Killing all running tasks in stage 226: Stage finished
17:38:16.322 INFO DAGScheduler - Job 169 finished: runJob at SparkHadoopWriter.scala:83, took 0.522306 s
17:38:16.322 INFO SparkHadoopWriter - Start to commit write Job job_202510271738159192766228709143977_1083.
17:38:16.323 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=listStatus src=/user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam.parts/_temporary/0 dst=null perm=null proto=rpc
17:38:16.323 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam.parts dst=null perm=null proto=rpc
17:38:16.324 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=listStatus src=/user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam.parts/_temporary/0/task_202510271738159192766228709143977_1083_r_000000 dst=null perm=null proto=rpc
17:38:16.324 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam.parts/.part-r-00000.bai dst=null perm=null proto=rpc
17:38:16.325 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=rename src=/user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam.parts/_temporary/0/task_202510271738159192766228709143977_1083_r_000000/.part-r-00000.bai dst=/user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam.parts/.part-r-00000.bai perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:16.325 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam.parts/part-r-00000 dst=null perm=null proto=rpc
17:38:16.326 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=rename src=/user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam.parts/_temporary/0/task_202510271738159192766228709143977_1083_r_000000/part-r-00000 dst=/user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam.parts/part-r-00000 perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:16.326 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=delete src=/user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam.parts/_temporary dst=null perm=null proto=rpc
17:38:16.327 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam.parts/_SUCCESS dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:16.328 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam.parts/_SUCCESS is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:38:16.328 INFO audit - allowed=false ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=delete src=/user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam.parts/.spark-staging-1083 dst=null perm=null proto=rpc
17:38:16.328 INFO SparkHadoopWriter - Write Job job_202510271738159192766228709143977_1083 committed. Elapsed time: 6 ms.
17:38:16.329 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam.parts/header dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:16.329 WARN IntelDeflaterFactory - Intel Deflater not supported, using Java.util.zip.Deflater
17:38:16.330 INFO StateChange - BLOCK* allocate blk_1073741887_1063, replicas=127.0.0.1:37927 for /user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam.parts/header
17:38:16.332 INFO DataNode - Receiving BP-617366194-10.1.0.252-1761586664431:blk_1073741887_1063 src: /127.0.0.1:58226 dest: /127.0.0.1:37927
17:38:16.333 INFO clienttrace - src: /127.0.0.1:58226, dest: /127.0.0.1:37927, bytes: 5901, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1369393814_1, offset: 0, srvID: 379dd5e5-59c3-42cd-b9d3-67ca5e44560b, blockid: BP-617366194-10.1.0.252-1761586664431:blk_1073741887_1063, duration(ns): 501839
17:38:16.334 INFO DataNode - PacketResponder: BP-617366194-10.1.0.252-1761586664431:blk_1073741887_1063, type=LAST_IN_PIPELINE terminating
17:38:16.334 INFO FSNamesystem - BLOCK* blk_1073741887_1063 is COMMITTED but not COMPLETE(numNodes= 0 < minimum = 1) in file /user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam.parts/header
17:38:16.735 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam.parts/header is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:38:16.737 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam.parts/terminator dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:16.738 INFO StateChange - BLOCK* allocate blk_1073741888_1064, replicas=127.0.0.1:37927 for /user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam.parts/terminator
17:38:16.739 INFO DataNode - Receiving BP-617366194-10.1.0.252-1761586664431:blk_1073741888_1064 src: /127.0.0.1:58228 dest: /127.0.0.1:37927
17:38:16.740 INFO clienttrace - src: /127.0.0.1:58228, dest: /127.0.0.1:37927, bytes: 28, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1369393814_1, offset: 0, srvID: 379dd5e5-59c3-42cd-b9d3-67ca5e44560b, blockid: BP-617366194-10.1.0.252-1761586664431:blk_1073741888_1064, duration(ns): 512863
17:38:16.740 INFO DataNode - PacketResponder: BP-617366194-10.1.0.252-1761586664431:blk_1073741888_1064, type=LAST_IN_PIPELINE terminating
17:38:16.741 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam.parts/terminator is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:38:16.742 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=listStatus src=/user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam.parts dst=null perm=null proto=rpc
17:38:16.743 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam.parts/output dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:16.743 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam.parts/output is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:38:16.744 INFO HadoopFileSystemWrapper - Concatenating 3 parts to hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam
17:38:16.744 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=concat src=[/user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam.parts/header, /user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam.parts/part-r-00000, /user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam.parts/terminator] dst=/user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam.parts/output perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:16.744 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam dst=null perm=null proto=rpc
17:38:16.745 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=delete src=/user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam dst=null perm=null proto=rpc
17:38:16.746 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=rename src=/user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam.parts/output dst=/user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:16.746 INFO HadoopFileSystemWrapper - Concatenating to hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam done
17:38:16.746 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam dst=null perm=null proto=rpc
17:38:16.746 INFO IndexFileMerger - Merging .bai files in temp directory hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam.parts/ to hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam.bai
17:38:16.747 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=listStatus src=/user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam.parts dst=null perm=null proto=rpc
17:38:16.747 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam.bai dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:16.748 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam.parts/.part-r-00000.bai dst=null perm=null proto=rpc
17:38:16.749 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam.parts/.part-r-00000.bai dst=null perm=null proto=rpc
17:38:16.750 WARN DFSUtil - Unexpected value for data transfer bytes=5516 duration=0
17:38:16.750 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=delete src=/user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam.parts/.part-r-00000.bai dst=null perm=null proto=rpc
17:38:16.751 INFO StateChange - BLOCK* allocate blk_1073741889_1065, replicas=127.0.0.1:37927 for /user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam.bai
17:38:16.752 INFO DataNode - Receiving BP-617366194-10.1.0.252-1761586664431:blk_1073741889_1065 src: /127.0.0.1:58236 dest: /127.0.0.1:37927
17:38:16.753 INFO clienttrace - src: /127.0.0.1:58236, dest: /127.0.0.1:37927, bytes: 5472, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1369393814_1, offset: 0, srvID: 379dd5e5-59c3-42cd-b9d3-67ca5e44560b, blockid: BP-617366194-10.1.0.252-1761586664431:blk_1073741889_1065, duration(ns): 553169
17:38:16.753 INFO DataNode - PacketResponder: BP-617366194-10.1.0.252-1761586664431:blk_1073741889_1065, type=LAST_IN_PIPELINE terminating
17:38:16.754 INFO FSNamesystem - BLOCK* blk_1073741889_1065 is COMMITTED but not COMPLETE(numNodes= 0 < minimum = 1) in file /user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam.bai
17:38:17.155 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam.bai is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:38:17.155 INFO IndexFileMerger - Done merging .bai files
17:38:17.156 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=delete src=/user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam.parts dst=null perm=null proto=rpc
17:38:17.165 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam.bai dst=null perm=null proto=rpc
17:38:17.165 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam dst=null perm=null proto=rpc
17:38:17.166 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam dst=null perm=null proto=rpc
17:38:17.166 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam dst=null perm=null proto=rpc
17:38:17.167 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam dst=null perm=null proto=rpc
17:38:17.167 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam.bai dst=null perm=null proto=rpc
17:38:17.168 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam.bai dst=null perm=null proto=rpc
17:38:17.168 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam.bai dst=null perm=null proto=rpc
17:38:17.169 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:17.170 WARN DFSUtil - Unexpected value for data transfer bytes=5949 duration=0
17:38:17.171 WARN DFSUtil - Unexpected value for data transfer bytes=5516 duration=0
17:38:17.172 WARN DFSUtil - Unexpected value for data transfer bytes=5516 duration=0
17:38:17.172 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam.sbi dst=null perm=null proto=rpc
17:38:17.174 INFO MemoryStore - Block broadcast_455 stored as values in memory (estimated size 297.9 KiB, free 1917.7 MiB)
17:38:17.179 INFO MemoryStore - Block broadcast_455_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1917.7 MiB)
17:38:17.179 INFO BlockManagerInfo - Added broadcast_455_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.6 MiB)
17:38:17.180 INFO SparkContext - Created broadcast 455 from newAPIHadoopFile at PathSplitSource.java:96
17:38:17.199 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam dst=null perm=null proto=rpc
17:38:17.199 INFO FileInputFormat - Total input files to process : 1
17:38:17.199 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam dst=null perm=null proto=rpc
17:38:17.232 INFO SparkContext - Starting job: collect at ReadsSparkSinkUnitTest.java:182
17:38:17.232 INFO DAGScheduler - Got job 170 (collect at ReadsSparkSinkUnitTest.java:182) with 1 output partitions
17:38:17.232 INFO DAGScheduler - Final stage: ResultStage 227 (collect at ReadsSparkSinkUnitTest.java:182)
17:38:17.232 INFO DAGScheduler - Parents of final stage: List()
17:38:17.232 INFO DAGScheduler - Missing parents: List()
17:38:17.232 INFO DAGScheduler - Submitting ResultStage 227 (MapPartitionsRDD[1090] at filter at ReadsSparkSource.java:96), which has no missing parents
17:38:17.247 INFO MemoryStore - Block broadcast_456 stored as values in memory (estimated size 426.2 KiB, free 1917.3 MiB)
17:38:17.251 INFO BlockManagerInfo - Removed broadcast_452_piece0 on localhost:37953 in memory (size: 9.6 KiB, free: 1919.6 MiB)
17:38:17.251 INFO BlockManagerInfo - Removed broadcast_451_piece0 on localhost:37953 in memory (size: 9.6 KiB, free: 1919.6 MiB)
17:38:17.251 INFO MemoryStore - Block broadcast_456_piece0 stored as bytes in memory (estimated size 153.6 KiB, free 1917.3 MiB)
17:38:17.251 INFO BlockManagerInfo - Added broadcast_456_piece0 in memory on localhost:37953 (size: 153.6 KiB, free: 1919.5 MiB)
17:38:17.252 INFO SparkContext - Created broadcast 456 from broadcast at DAGScheduler.scala:1580
17:38:17.252 INFO BlockManagerInfo - Removed broadcast_453_piece0 on localhost:37953 in memory (size: 166.1 KiB, free: 1919.6 MiB)
17:38:17.252 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 227 (MapPartitionsRDD[1090] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:38:17.252 INFO TaskSchedulerImpl - Adding task set 227.0 with 1 tasks resource profile 0
17:38:17.252 INFO TaskSetManager - Starting task 0.0 in stage 227.0 (TID 283) (localhost, executor driver, partition 0, ANY, 7852 bytes)
17:38:17.253 INFO BlockManagerInfo - Removed broadcast_450_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.7 MiB)
17:38:17.253 INFO Executor - Running task 0.0 in stage 227.0 (TID 283)
17:38:17.253 INFO BlockManagerInfo - Removed broadcast_454_piece0 on localhost:37953 in memory (size: 67.1 KiB, free: 1919.8 MiB)
17:38:17.284 INFO NewHadoopRDD - Input split: hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam:0+238969
17:38:17.285 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam dst=null perm=null proto=rpc
17:38:17.285 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam dst=null perm=null proto=rpc
17:38:17.286 WARN DFSUtil - Unexpected value for data transfer bytes=5949 duration=0
17:38:17.287 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam dst=null perm=null proto=rpc
17:38:17.287 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam dst=null perm=null proto=rpc
17:38:17.288 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam.bai dst=null perm=null proto=rpc
17:38:17.289 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam.bai dst=null perm=null proto=rpc
17:38:17.289 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam.bai dst=null perm=null proto=rpc
17:38:17.290 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:17.291 WARN DFSUtil - Unexpected value for data transfer bytes=5949 duration=0
17:38:17.293 WARN DFSUtil - Unexpected value for data transfer bytes=5516 duration=0
17:38:17.293 WARN DFSUtil - Unexpected value for data transfer bytes=5516 duration=0
17:38:17.294 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam dst=null perm=null proto=rpc
17:38:17.294 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam dst=null perm=null proto=rpc
17:38:17.294 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:17.295 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:17.295 WARN DFSUtil - Unexpected value for data transfer bytes=5949 duration=0
17:38:17.299 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:17.300 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:17.301 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:17.301 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:17.302 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:17.303 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:17.303 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:17.304 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:17.305 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:17.306 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:17.307 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:17.307 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:17.308 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:17.309 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:17.310 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:17.311 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:17.312 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:17.312 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:17.313 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:17.314 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:17.314 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:17.315 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:17.316 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:17.317 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:17.318 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:17.318 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:17.319 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:17.320 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:17.321 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:17.321 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:17.322 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:17.323 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:17.324 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:17.325 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:17.326 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:17.327 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:17.328 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:17.330 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:17.331 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:17.332 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:17.333 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:17.334 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:17.334 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:17.335 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:17.336 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:17.337 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:17.337 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:17.338 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:17.339 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:17.339 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:17.340 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:17.341 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:17.341 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:17.342 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:17.344 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:17.345 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:17.346 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:17.346 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:17.347 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:17.347 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:17.347 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam dst=null perm=null proto=rpc
17:38:17.348 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam dst=null perm=null proto=rpc
17:38:17.349 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam.bai dst=null perm=null proto=rpc
17:38:17.349 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam.bai dst=null perm=null proto=rpc
17:38:17.350 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam.bai dst=null perm=null proto=rpc
17:38:17.351 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:17.351 WARN DFSUtil - Unexpected value for data transfer bytes=5949 duration=0
17:38:17.353 WARN DFSUtil - Unexpected value for data transfer bytes=5516 duration=0
17:38:17.354 WARN DFSUtil - Unexpected value for data transfer bytes=5516 duration=0
17:38:17.355 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:17.355 WARN DFSUtil - Unexpected value for data transfer bytes=32 duration=0
17:38:17.358 INFO Executor - Finished task 0.0 in stage 227.0 (TID 283). 651483 bytes result sent to driver
17:38:17.359 INFO TaskSetManager - Finished task 0.0 in stage 227.0 (TID 283) in 107 ms on localhost (executor driver) (1/1)
17:38:17.359 INFO TaskSchedulerImpl - Removed TaskSet 227.0, whose tasks have all completed, from pool
17:38:17.360 INFO DAGScheduler - ResultStage 227 (collect at ReadsSparkSinkUnitTest.java:182) finished in 0.128 s
17:38:17.360 INFO DAGScheduler - Job 170 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:17.360 INFO TaskSchedulerImpl - Killing all running tasks in stage 227: Stage finished
17:38:17.360 INFO DAGScheduler - Job 170 finished: collect at ReadsSparkSinkUnitTest.java:182, took 0.128395 s
17:38:17.370 INFO SparkContext - Starting job: count at ReadsSparkSinkUnitTest.java:185
17:38:17.370 INFO DAGScheduler - Got job 171 (count at ReadsSparkSinkUnitTest.java:185) with 1 output partitions
17:38:17.370 INFO DAGScheduler - Final stage: ResultStage 228 (count at ReadsSparkSinkUnitTest.java:185)
17:38:17.370 INFO DAGScheduler - Parents of final stage: List()
17:38:17.370 INFO DAGScheduler - Missing parents: List()
17:38:17.370 INFO DAGScheduler - Submitting ResultStage 228 (MapPartitionsRDD[1071] at filter at ReadsSparkSource.java:96), which has no missing parents
17:38:17.386 INFO MemoryStore - Block broadcast_457 stored as values in memory (estimated size 426.1 KiB, free 1918.3 MiB)
17:38:17.387 INFO MemoryStore - Block broadcast_457_piece0 stored as bytes in memory (estimated size 153.6 KiB, free 1918.2 MiB)
17:38:17.387 INFO BlockManagerInfo - Added broadcast_457_piece0 in memory on localhost:37953 (size: 153.6 KiB, free: 1919.6 MiB)
17:38:17.387 INFO SparkContext - Created broadcast 457 from broadcast at DAGScheduler.scala:1580
17:38:17.387 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 228 (MapPartitionsRDD[1071] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:38:17.387 INFO TaskSchedulerImpl - Adding task set 228.0 with 1 tasks resource profile 0
17:38:17.388 INFO TaskSetManager - Starting task 0.0 in stage 228.0 (TID 284) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7872 bytes)
17:38:17.388 INFO Executor - Running task 0.0 in stage 228.0 (TID 284)
17:38:17.419 INFO NewHadoopRDD - Input split: file:/home/runner/work/gatk/gatk/src/test/resources/org/broadinstitute/hellbender/tools/BQSR/HiSeq.1mb.1RG.2k_lines.bam:0+222075
17:38:17.420 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:17.421 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:17.428 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:17.431 INFO Executor - Finished task 0.0 in stage 228.0 (TID 284). 989 bytes result sent to driver
17:38:17.432 INFO TaskSetManager - Finished task 0.0 in stage 228.0 (TID 284) in 44 ms on localhost (executor driver) (1/1)
17:38:17.432 INFO TaskSchedulerImpl - Removed TaskSet 228.0, whose tasks have all completed, from pool
17:38:17.432 INFO DAGScheduler - ResultStage 228 (count at ReadsSparkSinkUnitTest.java:185) finished in 0.062 s
17:38:17.432 INFO DAGScheduler - Job 171 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:17.432 INFO TaskSchedulerImpl - Killing all running tasks in stage 228: Stage finished
17:38:17.432 INFO DAGScheduler - Job 171 finished: count at ReadsSparkSinkUnitTest.java:185, took 0.061992 s
17:38:17.435 INFO SparkContext - Starting job: count at ReadsSparkSinkUnitTest.java:185
17:38:17.435 INFO DAGScheduler - Got job 172 (count at ReadsSparkSinkUnitTest.java:185) with 1 output partitions
17:38:17.435 INFO DAGScheduler - Final stage: ResultStage 229 (count at ReadsSparkSinkUnitTest.java:185)
17:38:17.435 INFO DAGScheduler - Parents of final stage: List()
17:38:17.435 INFO DAGScheduler - Missing parents: List()
17:38:17.435 INFO DAGScheduler - Submitting ResultStage 229 (MapPartitionsRDD[1090] at filter at ReadsSparkSource.java:96), which has no missing parents
17:38:17.450 INFO MemoryStore - Block broadcast_458 stored as values in memory (estimated size 426.1 KiB, free 1917.8 MiB)
17:38:17.452 INFO MemoryStore - Block broadcast_458_piece0 stored as bytes in memory (estimated size 153.6 KiB, free 1917.6 MiB)
17:38:17.452 INFO BlockManagerInfo - Added broadcast_458_piece0 in memory on localhost:37953 (size: 153.6 KiB, free: 1919.5 MiB)
17:38:17.452 INFO SparkContext - Created broadcast 458 from broadcast at DAGScheduler.scala:1580
17:38:17.452 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 229 (MapPartitionsRDD[1090] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:38:17.452 INFO TaskSchedulerImpl - Adding task set 229.0 with 1 tasks resource profile 0
17:38:17.453 INFO TaskSetManager - Starting task 0.0 in stage 229.0 (TID 285) (localhost, executor driver, partition 0, ANY, 7852 bytes)
17:38:17.453 INFO Executor - Running task 0.0 in stage 229.0 (TID 285)
17:38:17.483 INFO NewHadoopRDD - Input split: hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam:0+238969
17:38:17.484 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam dst=null perm=null proto=rpc
17:38:17.485 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam dst=null perm=null proto=rpc
17:38:17.486 WARN DFSUtil - Unexpected value for data transfer bytes=5949 duration=0
17:38:17.487 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam dst=null perm=null proto=rpc
17:38:17.487 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam dst=null perm=null proto=rpc
17:38:17.489 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam.bai dst=null perm=null proto=rpc
17:38:17.490 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam.bai dst=null perm=null proto=rpc
17:38:17.490 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam.bai dst=null perm=null proto=rpc
17:38:17.491 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:17.492 WARN DFSUtil - Unexpected value for data transfer bytes=5949 duration=0
17:38:17.494 WARN DFSUtil - Unexpected value for data transfer bytes=5516 duration=0
17:38:17.494 WARN DFSUtil - Unexpected value for data transfer bytes=5516 duration=0
17:38:17.494 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam dst=null perm=null proto=rpc
17:38:17.495 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam dst=null perm=null proto=rpc
17:38:17.495 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:17.496 WARN DFSUtil - Unexpected value for data transfer bytes=5949 duration=0
17:38:17.500 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:17.501 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:17.501 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:17.502 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:17.503 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:17.504 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:17.505 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:17.505 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:17.506 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:17.507 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:17.508 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:17.509 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:17.510 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:17.510 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:17.511 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:17.512 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:17.512 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:17.513 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:17.514 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:17.515 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:17.516 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:17.516 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:17.517 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:17.518 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:17.518 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:17.519 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:17.520 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:17.520 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:17.521 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:17.522 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:17.523 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:17.524 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:17.524 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:17.525 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:17.526 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:17.527 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:17.527 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:17.528 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:17.529 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:17.530 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:17.531 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:17.531 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:17.532 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:17.533 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:17.533 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:17.534 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:17.535 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:17.535 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:17.536 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:17.537 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:17.538 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:17.538 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:17.539 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:17.540 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:17.541 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:17.541 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:17.542 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:17.543 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:17.544 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:17.545 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:17.545 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:17.546 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:17.547 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:17.547 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam dst=null perm=null proto=rpc
17:38:17.548 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam dst=null perm=null proto=rpc
17:38:17.548 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam.bai dst=null perm=null proto=rpc
17:38:17.549 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam.bai dst=null perm=null proto=rpc
17:38:17.550 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_8d375a58-a5e9-4c08-a876-a95dfa349b3a.bam.bai dst=null perm=null proto=rpc
17:38:17.551 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:17.552 WARN DFSUtil - Unexpected value for data transfer bytes=5949 duration=0
17:38:17.553 WARN DFSUtil - Unexpected value for data transfer bytes=5516 duration=0
17:38:17.554 WARN DFSUtil - Unexpected value for data transfer bytes=5516 duration=0
17:38:17.555 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:17.556 WARN DFSUtil - Unexpected value for data transfer bytes=32 duration=0
17:38:17.557 INFO Executor - Finished task 0.0 in stage 229.0 (TID 285). 989 bytes result sent to driver
17:38:17.558 INFO TaskSetManager - Finished task 0.0 in stage 229.0 (TID 285) in 106 ms on localhost (executor driver) (1/1)
17:38:17.558 INFO TaskSchedulerImpl - Removed TaskSet 229.0, whose tasks have all completed, from pool
17:38:17.558 INFO DAGScheduler - ResultStage 229 (count at ReadsSparkSinkUnitTest.java:185) finished in 0.123 s
17:38:17.558 INFO DAGScheduler - Job 172 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:17.558 INFO TaskSchedulerImpl - Killing all running tasks in stage 229: Stage finished
17:38:17.558 INFO DAGScheduler - Job 172 finished: count at ReadsSparkSinkUnitTest.java:185, took 0.123106 s
17:38:17.566 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_b685c02d-74bf-4b11-88a9-99b561f1ae4a.bam dst=null perm=null proto=rpc
17:38:17.567 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest1_b685c02d-74bf-4b11-88a9-99b561f1ae4a.bam dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:17.568 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest1_b685c02d-74bf-4b11-88a9-99b561f1ae4a.bam is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:38:17.568 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_b685c02d-74bf-4b11-88a9-99b561f1ae4a.bam dst=null perm=null proto=rpc
17:38:17.569 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:17.571 INFO MemoryStore - Block broadcast_459 stored as values in memory (estimated size 297.9 KiB, free 1917.3 MiB)
17:38:17.576 INFO MemoryStore - Block broadcast_459_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1917.3 MiB)
17:38:17.577 INFO BlockManagerInfo - Added broadcast_459_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.4 MiB)
17:38:17.577 INFO SparkContext - Created broadcast 459 from newAPIHadoopFile at PathSplitSource.java:96
17:38:17.595 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:17.597 INFO MemoryStore - Block broadcast_460 stored as values in memory (estimated size 297.9 KiB, free 1917.0 MiB)
17:38:17.603 INFO MemoryStore - Block broadcast_460_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1916.9 MiB)
17:38:17.603 INFO BlockManagerInfo - Added broadcast_460_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.4 MiB)
17:38:17.603 INFO SparkContext - Created broadcast 460 from newAPIHadoopFile at PathSplitSource.java:96
17:38:17.622 INFO FileInputFormat - Total input files to process : 1
17:38:17.623 INFO MemoryStore - Block broadcast_461 stored as values in memory (estimated size 160.7 KiB, free 1916.8 MiB)
17:38:17.624 INFO MemoryStore - Block broadcast_461_piece0 stored as bytes in memory (estimated size 9.6 KiB, free 1916.8 MiB)
17:38:17.624 INFO BlockManagerInfo - Added broadcast_461_piece0 in memory on localhost:37953 (size: 9.6 KiB, free: 1919.3 MiB)
17:38:17.624 INFO SparkContext - Created broadcast 461 from broadcast at ReadsSparkSink.java:133
17:38:17.625 WARN HtsjdkReadsRddStorage - Unrecognized write option: DISABLE
17:38:17.626 INFO MemoryStore - Block broadcast_462 stored as values in memory (estimated size 163.2 KiB, free 1916.6 MiB)
17:38:17.629 INFO MemoryStore - Block broadcast_462_piece0 stored as bytes in memory (estimated size 9.6 KiB, free 1916.6 MiB)
17:38:17.629 INFO BlockManagerInfo - Removed broadcast_458_piece0 on localhost:37953 in memory (size: 153.6 KiB, free: 1919.5 MiB)
17:38:17.629 INFO BlockManagerInfo - Added broadcast_462_piece0 in memory on localhost:37953 (size: 9.6 KiB, free: 1919.5 MiB)
17:38:17.629 INFO SparkContext - Created broadcast 462 from broadcast at BamSink.java:76
17:38:17.630 INFO BlockManagerInfo - Removed broadcast_449_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.5 MiB)
17:38:17.631 INFO BlockManagerInfo - Removed broadcast_460_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.6 MiB)
17:38:17.631 INFO BlockManagerInfo - Removed broadcast_455_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.6 MiB)
17:38:17.631 INFO BlockManagerInfo - Removed broadcast_456_piece0 on localhost:37953 in memory (size: 153.6 KiB, free: 1919.8 MiB)
17:38:17.632 INFO BlockManagerInfo - Removed broadcast_457_piece0 on localhost:37953 in memory (size: 153.6 KiB, free: 1919.9 MiB)
17:38:17.632 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_b685c02d-74bf-4b11-88a9-99b561f1ae4a.bam.parts dst=null perm=null proto=rpc
17:38:17.632 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:17.632 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:17.632 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:17.633 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=mkdirs src=/user/runner/ReadsSparkSinkUnitTest1_b685c02d-74bf-4b11-88a9-99b561f1ae4a.bam.parts/_temporary/0 dst=null perm=runner:supergroup:rwxr-xr-x proto=rpc
17:38:17.639 INFO SparkContext - Starting job: runJob at SparkHadoopWriter.scala:83
17:38:17.639 INFO DAGScheduler - Registering RDD 1104 (mapToPair at SparkUtils.java:161) as input to shuffle 46
17:38:17.639 INFO DAGScheduler - Got job 173 (runJob at SparkHadoopWriter.scala:83) with 1 output partitions
17:38:17.639 INFO DAGScheduler - Final stage: ResultStage 231 (runJob at SparkHadoopWriter.scala:83)
17:38:17.639 INFO DAGScheduler - Parents of final stage: List(ShuffleMapStage 230)
17:38:17.639 INFO DAGScheduler - Missing parents: List(ShuffleMapStage 230)
17:38:17.639 INFO DAGScheduler - Submitting ShuffleMapStage 230 (MapPartitionsRDD[1104] at mapToPair at SparkUtils.java:161), which has no missing parents
17:38:17.655 INFO MemoryStore - Block broadcast_463 stored as values in memory (estimated size 520.4 KiB, free 1918.8 MiB)
17:38:17.656 INFO MemoryStore - Block broadcast_463_piece0 stored as bytes in memory (estimated size 166.1 KiB, free 1918.7 MiB)
17:38:17.656 INFO BlockManagerInfo - Added broadcast_463_piece0 in memory on localhost:37953 (size: 166.1 KiB, free: 1919.8 MiB)
17:38:17.656 INFO SparkContext - Created broadcast 463 from broadcast at DAGScheduler.scala:1580
17:38:17.656 INFO DAGScheduler - Submitting 1 missing tasks from ShuffleMapStage 230 (MapPartitionsRDD[1104] at mapToPair at SparkUtils.java:161) (first 15 tasks are for partitions Vector(0))
17:38:17.656 INFO TaskSchedulerImpl - Adding task set 230.0 with 1 tasks resource profile 0
17:38:17.657 INFO TaskSetManager - Starting task 0.0 in stage 230.0 (TID 286) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7861 bytes)
17:38:17.657 INFO Executor - Running task 0.0 in stage 230.0 (TID 286)
17:38:17.689 INFO NewHadoopRDD - Input split: file:/home/runner/work/gatk/gatk/src/test/resources/org/broadinstitute/hellbender/tools/BQSR/HiSeq.1mb.1RG.2k_lines.bam:0+222075
17:38:17.690 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:17.691 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:17.698 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:17.706 INFO Executor - Finished task 0.0 in stage 230.0 (TID 286). 1148 bytes result sent to driver
17:38:17.707 INFO TaskSetManager - Finished task 0.0 in stage 230.0 (TID 286) in 50 ms on localhost (executor driver) (1/1)
17:38:17.707 INFO TaskSchedulerImpl - Removed TaskSet 230.0, whose tasks have all completed, from pool
17:38:17.707 INFO DAGScheduler - ShuffleMapStage 230 (mapToPair at SparkUtils.java:161) finished in 0.067 s
17:38:17.707 INFO DAGScheduler - looking for newly runnable stages
17:38:17.707 INFO DAGScheduler - running: HashSet()
17:38:17.707 INFO DAGScheduler - waiting: HashSet(ResultStage 231)
17:38:17.707 INFO DAGScheduler - failed: HashSet()
17:38:17.707 INFO DAGScheduler - Submitting ResultStage 231 (MapPartitionsRDD[1109] at mapToPair at BamSink.java:91), which has no missing parents
17:38:17.714 INFO MemoryStore - Block broadcast_464 stored as values in memory (estimated size 241.5 KiB, free 1918.4 MiB)
17:38:17.714 INFO MemoryStore - Block broadcast_464_piece0 stored as bytes in memory (estimated size 67.1 KiB, free 1918.4 MiB)
17:38:17.714 INFO BlockManagerInfo - Added broadcast_464_piece0 in memory on localhost:37953 (size: 67.1 KiB, free: 1919.7 MiB)
17:38:17.714 INFO SparkContext - Created broadcast 464 from broadcast at DAGScheduler.scala:1580
17:38:17.715 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 231 (MapPartitionsRDD[1109] at mapToPair at BamSink.java:91) (first 15 tasks are for partitions Vector(0))
17:38:17.715 INFO TaskSchedulerImpl - Adding task set 231.0 with 1 tasks resource profile 0
17:38:17.715 INFO TaskSetManager - Starting task 0.0 in stage 231.0 (TID 287) (localhost, executor driver, partition 0, NODE_LOCAL, 7513 bytes)
17:38:17.715 INFO Executor - Running task 0.0 in stage 231.0 (TID 287)
17:38:17.720 INFO ShuffleBlockFetcherIterator - Getting 1 (343.8 KiB) non-empty blocks including 1 (343.8 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
17:38:17.720 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
17:38:17.731 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:17.731 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:17.731 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:17.732 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:17.732 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:17.732 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:17.733 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest1_b685c02d-74bf-4b11-88a9-99b561f1ae4a.bam.parts/_temporary/0/_temporary/attempt_20251027173817327816144223674985_1109_r_000000_0/part-r-00000 dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:17.733 WARN IntelDeflaterFactory - Intel Deflater not supported, using Java.util.zip.Deflater
17:38:17.734 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest1_b685c02d-74bf-4b11-88a9-99b561f1ae4a.bam.parts/_temporary/0/_temporary/attempt_20251027173817327816144223674985_1109_r_000000_0/.part-r-00000.sbi dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:17.737 INFO StateChange - BLOCK* allocate blk_1073741890_1066, replicas=127.0.0.1:37927 for /user/runner/ReadsSparkSinkUnitTest1_b685c02d-74bf-4b11-88a9-99b561f1ae4a.bam.parts/_temporary/0/_temporary/attempt_20251027173817327816144223674985_1109_r_000000_0/part-r-00000
17:38:17.739 INFO DataNode - Receiving BP-617366194-10.1.0.252-1761586664431:blk_1073741890_1066 src: /127.0.0.1:48874 dest: /127.0.0.1:37927
17:38:17.743 INFO clienttrace - src: /127.0.0.1:48874, dest: /127.0.0.1:37927, bytes: 233040, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1369393814_1, offset: 0, srvID: 379dd5e5-59c3-42cd-b9d3-67ca5e44560b, blockid: BP-617366194-10.1.0.252-1761586664431:blk_1073741890_1066, duration(ns): 3295152
17:38:17.743 INFO DataNode - PacketResponder: BP-617366194-10.1.0.252-1761586664431:blk_1073741890_1066, type=LAST_IN_PIPELINE terminating
17:38:17.743 INFO FSNamesystem - BLOCK* blk_1073741890_1066 is COMMITTED but not COMPLETE(numNodes= 0 < minimum = 1) in file /user/runner/ReadsSparkSinkUnitTest1_b685c02d-74bf-4b11-88a9-99b561f1ae4a.bam.parts/_temporary/0/_temporary/attempt_20251027173817327816144223674985_1109_r_000000_0/part-r-00000
17:38:18.144 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest1_b685c02d-74bf-4b11-88a9-99b561f1ae4a.bam.parts/_temporary/0/_temporary/attempt_20251027173817327816144223674985_1109_r_000000_0/part-r-00000 is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:38:18.145 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_b685c02d-74bf-4b11-88a9-99b561f1ae4a.bam.parts/_temporary/0/_temporary/attempt_20251027173817327816144223674985_1109_r_000000_0/part-r-00000 dst=null perm=null proto=rpc
17:38:18.146 INFO StateChange - BLOCK* allocate blk_1073741891_1067, replicas=127.0.0.1:37927 for /user/runner/ReadsSparkSinkUnitTest1_b685c02d-74bf-4b11-88a9-99b561f1ae4a.bam.parts/_temporary/0/_temporary/attempt_20251027173817327816144223674985_1109_r_000000_0/.part-r-00000.sbi
17:38:18.147 INFO DataNode - Receiving BP-617366194-10.1.0.252-1761586664431:blk_1073741891_1067 src: /127.0.0.1:48884 dest: /127.0.0.1:37927
17:38:18.148 INFO clienttrace - src: /127.0.0.1:48884, dest: /127.0.0.1:37927, bytes: 212, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1369393814_1, offset: 0, srvID: 379dd5e5-59c3-42cd-b9d3-67ca5e44560b, blockid: BP-617366194-10.1.0.252-1761586664431:blk_1073741891_1067, duration(ns): 497374
17:38:18.148 INFO DataNode - PacketResponder: BP-617366194-10.1.0.252-1761586664431:blk_1073741891_1067, type=LAST_IN_PIPELINE terminating
17:38:18.149 INFO FSNamesystem - BLOCK* blk_1073741891_1067 is COMMITTED but not COMPLETE(numNodes= 0 < minimum = 1) in file /user/runner/ReadsSparkSinkUnitTest1_b685c02d-74bf-4b11-88a9-99b561f1ae4a.bam.parts/_temporary/0/_temporary/attempt_20251027173817327816144223674985_1109_r_000000_0/.part-r-00000.sbi
17:38:18.549 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest1_b685c02d-74bf-4b11-88a9-99b561f1ae4a.bam.parts/_temporary/0/_temporary/attempt_20251027173817327816144223674985_1109_r_000000_0/.part-r-00000.sbi is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:38:18.551 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_b685c02d-74bf-4b11-88a9-99b561f1ae4a.bam.parts/_temporary/0/_temporary/attempt_20251027173817327816144223674985_1109_r_000000_0 dst=null perm=null proto=rpc
17:38:18.551 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_b685c02d-74bf-4b11-88a9-99b561f1ae4a.bam.parts/_temporary/0/_temporary/attempt_20251027173817327816144223674985_1109_r_000000_0 dst=null perm=null proto=rpc
17:38:18.552 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_b685c02d-74bf-4b11-88a9-99b561f1ae4a.bam.parts/_temporary/0/task_20251027173817327816144223674985_1109_r_000000 dst=null perm=null proto=rpc
17:38:18.552 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=rename src=/user/runner/ReadsSparkSinkUnitTest1_b685c02d-74bf-4b11-88a9-99b561f1ae4a.bam.parts/_temporary/0/_temporary/attempt_20251027173817327816144223674985_1109_r_000000_0 dst=/user/runner/ReadsSparkSinkUnitTest1_b685c02d-74bf-4b11-88a9-99b561f1ae4a.bam.parts/_temporary/0/task_20251027173817327816144223674985_1109_r_000000 perm=runner:supergroup:rwxr-xr-x proto=rpc
17:38:18.552 INFO FileOutputCommitter - Saved output of task 'attempt_20251027173817327816144223674985_1109_r_000000_0' to hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest1_b685c02d-74bf-4b11-88a9-99b561f1ae4a.bam.parts/_temporary/0/task_20251027173817327816144223674985_1109_r_000000
17:38:18.553 INFO SparkHadoopMapRedUtil - attempt_20251027173817327816144223674985_1109_r_000000_0: Committed. Elapsed time: 1 ms.
17:38:18.553 INFO Executor - Finished task 0.0 in stage 231.0 (TID 287). 1858 bytes result sent to driver
17:38:18.553 INFO TaskSetManager - Finished task 0.0 in stage 231.0 (TID 287) in 838 ms on localhost (executor driver) (1/1)
17:38:18.553 INFO TaskSchedulerImpl - Removed TaskSet 231.0, whose tasks have all completed, from pool
17:38:18.554 INFO DAGScheduler - ResultStage 231 (runJob at SparkHadoopWriter.scala:83) finished in 0.846 s
17:38:18.554 INFO DAGScheduler - Job 173 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:18.554 INFO TaskSchedulerImpl - Killing all running tasks in stage 231: Stage finished
17:38:18.554 INFO DAGScheduler - Job 173 finished: runJob at SparkHadoopWriter.scala:83, took 0.915061 s
17:38:18.554 INFO SparkHadoopWriter - Start to commit write Job job_20251027173817327816144223674985_1109.
17:38:18.554 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=listStatus src=/user/runner/ReadsSparkSinkUnitTest1_b685c02d-74bf-4b11-88a9-99b561f1ae4a.bam.parts/_temporary/0 dst=null perm=null proto=rpc
17:38:18.555 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_b685c02d-74bf-4b11-88a9-99b561f1ae4a.bam.parts dst=null perm=null proto=rpc
17:38:18.556 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=listStatus src=/user/runner/ReadsSparkSinkUnitTest1_b685c02d-74bf-4b11-88a9-99b561f1ae4a.bam.parts/_temporary/0/task_20251027173817327816144223674985_1109_r_000000 dst=null perm=null proto=rpc
17:38:18.556 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_b685c02d-74bf-4b11-88a9-99b561f1ae4a.bam.parts/.part-r-00000.sbi dst=null perm=null proto=rpc
17:38:18.557 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=rename src=/user/runner/ReadsSparkSinkUnitTest1_b685c02d-74bf-4b11-88a9-99b561f1ae4a.bam.parts/_temporary/0/task_20251027173817327816144223674985_1109_r_000000/.part-r-00000.sbi dst=/user/runner/ReadsSparkSinkUnitTest1_b685c02d-74bf-4b11-88a9-99b561f1ae4a.bam.parts/.part-r-00000.sbi perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:18.557 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_b685c02d-74bf-4b11-88a9-99b561f1ae4a.bam.parts/part-r-00000 dst=null perm=null proto=rpc
17:38:18.558 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=rename src=/user/runner/ReadsSparkSinkUnitTest1_b685c02d-74bf-4b11-88a9-99b561f1ae4a.bam.parts/_temporary/0/task_20251027173817327816144223674985_1109_r_000000/part-r-00000 dst=/user/runner/ReadsSparkSinkUnitTest1_b685c02d-74bf-4b11-88a9-99b561f1ae4a.bam.parts/part-r-00000 perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:18.558 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=delete src=/user/runner/ReadsSparkSinkUnitTest1_b685c02d-74bf-4b11-88a9-99b561f1ae4a.bam.parts/_temporary dst=null perm=null proto=rpc
17:38:18.559 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest1_b685c02d-74bf-4b11-88a9-99b561f1ae4a.bam.parts/_SUCCESS dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:18.560 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest1_b685c02d-74bf-4b11-88a9-99b561f1ae4a.bam.parts/_SUCCESS is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:38:18.560 INFO audit - allowed=false ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=delete src=/user/runner/ReadsSparkSinkUnitTest1_b685c02d-74bf-4b11-88a9-99b561f1ae4a.bam.parts/.spark-staging-1109 dst=null perm=null proto=rpc
17:38:18.561 INFO SparkHadoopWriter - Write Job job_20251027173817327816144223674985_1109 committed. Elapsed time: 6 ms.
17:38:18.561 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest1_b685c02d-74bf-4b11-88a9-99b561f1ae4a.bam.parts/header dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:18.561 WARN IntelDeflaterFactory - Intel Deflater not supported, using Java.util.zip.Deflater
17:38:18.563 INFO StateChange - BLOCK* allocate blk_1073741892_1068, replicas=127.0.0.1:37927 for /user/runner/ReadsSparkSinkUnitTest1_b685c02d-74bf-4b11-88a9-99b561f1ae4a.bam.parts/header
17:38:18.563 INFO DataNode - Receiving BP-617366194-10.1.0.252-1761586664431:blk_1073741892_1068 src: /127.0.0.1:48896 dest: /127.0.0.1:37927
17:38:18.565 INFO clienttrace - src: /127.0.0.1:48896, dest: /127.0.0.1:37927, bytes: 5901, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1369393814_1, offset: 0, srvID: 379dd5e5-59c3-42cd-b9d3-67ca5e44560b, blockid: BP-617366194-10.1.0.252-1761586664431:blk_1073741892_1068, duration(ns): 512655
17:38:18.565 INFO DataNode - PacketResponder: BP-617366194-10.1.0.252-1761586664431:blk_1073741892_1068, type=LAST_IN_PIPELINE terminating
17:38:18.565 INFO FSNamesystem - BLOCK* blk_1073741892_1068 is COMMITTED but not COMPLETE(numNodes= 0 < minimum = 1) in file /user/runner/ReadsSparkSinkUnitTest1_b685c02d-74bf-4b11-88a9-99b561f1ae4a.bam.parts/header
17:38:18.599 INFO FsDatasetAsyncDiskService - Scheduling blk_1073741879_1055 replica FinalizedReplica, blk_1073741879_1055, FINALIZED
getNumBytes() = 13492
getBytesOnDisk() = 13492
getVisibleLength()= 13492
getVolume() = /tmp/minicluster_storage14473397491642437130/data/data1
getBlockURI() = file:/tmp/minicluster_storage14473397491642437130/data/data1/current/BP-617366194-10.1.0.252-1761586664431/current/finalized/subdir0/subdir0/blk_1073741879 for deletion
17:38:18.599 INFO FsDatasetAsyncDiskService - Scheduling blk_1073741880_1056 replica FinalizedReplica, blk_1073741880_1056, FINALIZED
getNumBytes() = 5472
getBytesOnDisk() = 5472
getVisibleLength()= 5472
getVolume() = /tmp/minicluster_storage14473397491642437130/data/data2
getBlockURI() = file:/tmp/minicluster_storage14473397491642437130/data/data2/current/BP-617366194-10.1.0.252-1761586664431/current/finalized/subdir0/subdir0/blk_1073741880 for deletion
17:38:18.599 INFO FsDatasetAsyncDiskService - Scheduling blk_1073741886_1062 replica FinalizedReplica, blk_1073741886_1062, FINALIZED
getNumBytes() = 5472
getBytesOnDisk() = 5472
getVisibleLength()= 5472
getVolume() = /tmp/minicluster_storage14473397491642437130/data/data2
getBlockURI() = file:/tmp/minicluster_storage14473397491642437130/data/data2/current/BP-617366194-10.1.0.252-1761586664431/current/finalized/subdir0/subdir0/blk_1073741886 for deletion
17:38:18.599 INFO FsDatasetAsyncDiskService - Deleted BP-617366194-10.1.0.252-1761586664431 blk_1073741879_1055 URI file:/tmp/minicluster_storage14473397491642437130/data/data1/current/BP-617366194-10.1.0.252-1761586664431/current/finalized/subdir0/subdir0/blk_1073741879
17:38:18.599 INFO FsDatasetAsyncDiskService - Deleted BP-617366194-10.1.0.252-1761586664431 blk_1073741880_1056 URI file:/tmp/minicluster_storage14473397491642437130/data/data2/current/BP-617366194-10.1.0.252-1761586664431/current/finalized/subdir0/subdir0/blk_1073741880
17:38:18.599 INFO FsDatasetAsyncDiskService - Deleted BP-617366194-10.1.0.252-1761586664431 blk_1073741886_1062 URI file:/tmp/minicluster_storage14473397491642437130/data/data2/current/BP-617366194-10.1.0.252-1761586664431/current/finalized/subdir0/subdir0/blk_1073741886
17:38:18.966 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest1_b685c02d-74bf-4b11-88a9-99b561f1ae4a.bam.parts/header is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:38:18.967 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest1_b685c02d-74bf-4b11-88a9-99b561f1ae4a.bam.parts/terminator dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:18.968 INFO StateChange - BLOCK* allocate blk_1073741893_1069, replicas=127.0.0.1:37927 for /user/runner/ReadsSparkSinkUnitTest1_b685c02d-74bf-4b11-88a9-99b561f1ae4a.bam.parts/terminator
17:38:18.969 INFO DataNode - Receiving BP-617366194-10.1.0.252-1761586664431:blk_1073741893_1069 src: /127.0.0.1:48912 dest: /127.0.0.1:37927
17:38:18.971 INFO clienttrace - src: /127.0.0.1:48912, dest: /127.0.0.1:37927, bytes: 28, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1369393814_1, offset: 0, srvID: 379dd5e5-59c3-42cd-b9d3-67ca5e44560b, blockid: BP-617366194-10.1.0.252-1761586664431:blk_1073741893_1069, duration(ns): 496431
17:38:18.971 INFO DataNode - PacketResponder: BP-617366194-10.1.0.252-1761586664431:blk_1073741893_1069, type=LAST_IN_PIPELINE terminating
17:38:18.971 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest1_b685c02d-74bf-4b11-88a9-99b561f1ae4a.bam.parts/terminator is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:38:18.972 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=listStatus src=/user/runner/ReadsSparkSinkUnitTest1_b685c02d-74bf-4b11-88a9-99b561f1ae4a.bam.parts dst=null perm=null proto=rpc
17:38:18.973 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest1_b685c02d-74bf-4b11-88a9-99b561f1ae4a.bam.parts/output dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:18.973 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest1_b685c02d-74bf-4b11-88a9-99b561f1ae4a.bam.parts/output is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:38:18.974 INFO HadoopFileSystemWrapper - Concatenating 3 parts to hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest1_b685c02d-74bf-4b11-88a9-99b561f1ae4a.bam
17:38:18.974 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=concat src=[/user/runner/ReadsSparkSinkUnitTest1_b685c02d-74bf-4b11-88a9-99b561f1ae4a.bam.parts/header, /user/runner/ReadsSparkSinkUnitTest1_b685c02d-74bf-4b11-88a9-99b561f1ae4a.bam.parts/part-r-00000, /user/runner/ReadsSparkSinkUnitTest1_b685c02d-74bf-4b11-88a9-99b561f1ae4a.bam.parts/terminator] dst=/user/runner/ReadsSparkSinkUnitTest1_b685c02d-74bf-4b11-88a9-99b561f1ae4a.bam.parts/output perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:18.974 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_b685c02d-74bf-4b11-88a9-99b561f1ae4a.bam dst=null perm=null proto=rpc
17:38:18.975 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=delete src=/user/runner/ReadsSparkSinkUnitTest1_b685c02d-74bf-4b11-88a9-99b561f1ae4a.bam dst=null perm=null proto=rpc
17:38:18.976 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=rename src=/user/runner/ReadsSparkSinkUnitTest1_b685c02d-74bf-4b11-88a9-99b561f1ae4a.bam.parts/output dst=/user/runner/ReadsSparkSinkUnitTest1_b685c02d-74bf-4b11-88a9-99b561f1ae4a.bam perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:18.976 INFO HadoopFileSystemWrapper - Concatenating to hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest1_b685c02d-74bf-4b11-88a9-99b561f1ae4a.bam done
17:38:18.976 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_b685c02d-74bf-4b11-88a9-99b561f1ae4a.bam dst=null perm=null proto=rpc
17:38:18.976 INFO IndexFileMerger - Merging .sbi files in temp directory hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest1_b685c02d-74bf-4b11-88a9-99b561f1ae4a.bam.parts/ to hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest1_b685c02d-74bf-4b11-88a9-99b561f1ae4a.bam.sbi
17:38:18.976 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=listStatus src=/user/runner/ReadsSparkSinkUnitTest1_b685c02d-74bf-4b11-88a9-99b561f1ae4a.bam.parts dst=null perm=null proto=rpc
17:38:18.977 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest1_b685c02d-74bf-4b11-88a9-99b561f1ae4a.bam.sbi dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:18.978 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_b685c02d-74bf-4b11-88a9-99b561f1ae4a.bam.parts/.part-r-00000.sbi dst=null perm=null proto=rpc
17:38:18.979 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_b685c02d-74bf-4b11-88a9-99b561f1ae4a.bam.parts/.part-r-00000.sbi dst=null perm=null proto=rpc
17:38:18.980 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=delete src=/user/runner/ReadsSparkSinkUnitTest1_b685c02d-74bf-4b11-88a9-99b561f1ae4a.bam.parts/.part-r-00000.sbi dst=null perm=null proto=rpc
17:38:18.981 INFO StateChange - BLOCK* allocate blk_1073741894_1070, replicas=127.0.0.1:37927 for /user/runner/ReadsSparkSinkUnitTest1_b685c02d-74bf-4b11-88a9-99b561f1ae4a.bam.sbi
17:38:18.981 INFO DataNode - Receiving BP-617366194-10.1.0.252-1761586664431:blk_1073741894_1070 src: /127.0.0.1:48920 dest: /127.0.0.1:37927
17:38:18.982 INFO clienttrace - src: /127.0.0.1:48920, dest: /127.0.0.1:37927, bytes: 212, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1369393814_1, offset: 0, srvID: 379dd5e5-59c3-42cd-b9d3-67ca5e44560b, blockid: BP-617366194-10.1.0.252-1761586664431:blk_1073741894_1070, duration(ns): 490486
17:38:18.983 INFO DataNode - PacketResponder: BP-617366194-10.1.0.252-1761586664431:blk_1073741894_1070, type=LAST_IN_PIPELINE terminating
17:38:18.983 INFO FSNamesystem - BLOCK* blk_1073741894_1070 is COMMITTED but not COMPLETE(numNodes= 0 < minimum = 1) in file /user/runner/ReadsSparkSinkUnitTest1_b685c02d-74bf-4b11-88a9-99b561f1ae4a.bam.sbi
17:38:19.384 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest1_b685c02d-74bf-4b11-88a9-99b561f1ae4a.bam.sbi is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:38:19.385 INFO IndexFileMerger - Done merging .sbi files
17:38:19.385 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=delete src=/user/runner/ReadsSparkSinkUnitTest1_b685c02d-74bf-4b11-88a9-99b561f1ae4a.bam.parts dst=null perm=null proto=rpc
17:38:19.394 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_b685c02d-74bf-4b11-88a9-99b561f1ae4a.bam.sbi dst=null perm=null proto=rpc
17:38:19.394 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_b685c02d-74bf-4b11-88a9-99b561f1ae4a.bam.sbi dst=null perm=null proto=rpc
17:38:19.395 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_b685c02d-74bf-4b11-88a9-99b561f1ae4a.bam.sbi dst=null perm=null proto=rpc
17:38:19.396 WARN DFSUtil - Unexpected value for data transfer bytes=216 duration=0
17:38:19.396 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_b685c02d-74bf-4b11-88a9-99b561f1ae4a.bam dst=null perm=null proto=rpc
17:38:19.397 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_b685c02d-74bf-4b11-88a9-99b561f1ae4a.bam dst=null perm=null proto=rpc
17:38:19.397 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_b685c02d-74bf-4b11-88a9-99b561f1ae4a.bam dst=null perm=null proto=rpc
17:38:19.398 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_b685c02d-74bf-4b11-88a9-99b561f1ae4a.bam dst=null perm=null proto=rpc
17:38:19.399 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_b685c02d-74bf-4b11-88a9-99b561f1ae4a.bam.bai dst=null perm=null proto=rpc
17:38:19.399 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_b685c02d-74bf-4b11-88a9-99b561f1ae4a.bai dst=null perm=null proto=rpc
17:38:19.400 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:19.400 WARN DFSUtil - Unexpected value for data transfer bytes=5949 duration=0
17:38:19.401 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:19.402 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_b685c02d-74bf-4b11-88a9-99b561f1ae4a.bam.sbi dst=null perm=null proto=rpc
17:38:19.402 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_b685c02d-74bf-4b11-88a9-99b561f1ae4a.bam.sbi dst=null perm=null proto=rpc
17:38:19.403 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_b685c02d-74bf-4b11-88a9-99b561f1ae4a.bam.sbi dst=null perm=null proto=rpc
17:38:19.403 WARN DFSUtil - Unexpected value for data transfer bytes=216 duration=0
17:38:19.404 INFO MemoryStore - Block broadcast_465 stored as values in memory (estimated size 320.0 B, free 1918.4 MiB)
17:38:19.404 INFO MemoryStore - Block broadcast_465_piece0 stored as bytes in memory (estimated size 233.0 B, free 1918.4 MiB)
17:38:19.404 INFO BlockManagerInfo - Added broadcast_465_piece0 in memory on localhost:37953 (size: 233.0 B, free: 1919.7 MiB)
17:38:19.404 INFO SparkContext - Created broadcast 465 from broadcast at BamSource.java:104
17:38:19.405 INFO MemoryStore - Block broadcast_466 stored as values in memory (estimated size 297.9 KiB, free 1918.1 MiB)
17:38:19.411 INFO MemoryStore - Block broadcast_466_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1918.0 MiB)
17:38:19.411 INFO BlockManagerInfo - Added broadcast_466_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.7 MiB)
17:38:19.411 INFO SparkContext - Created broadcast 466 from newAPIHadoopFile at PathSplitSource.java:96
17:38:19.420 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_b685c02d-74bf-4b11-88a9-99b561f1ae4a.bam dst=null perm=null proto=rpc
17:38:19.420 INFO FileInputFormat - Total input files to process : 1
17:38:19.420 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_b685c02d-74bf-4b11-88a9-99b561f1ae4a.bam dst=null perm=null proto=rpc
17:38:19.433 INFO SparkContext - Starting job: collect at ReadsSparkSinkUnitTest.java:182
17:38:19.434 INFO DAGScheduler - Got job 174 (collect at ReadsSparkSinkUnitTest.java:182) with 1 output partitions
17:38:19.434 INFO DAGScheduler - Final stage: ResultStage 232 (collect at ReadsSparkSinkUnitTest.java:182)
17:38:19.434 INFO DAGScheduler - Parents of final stage: List()
17:38:19.434 INFO DAGScheduler - Missing parents: List()
17:38:19.434 INFO DAGScheduler - Submitting ResultStage 232 (MapPartitionsRDD[1115] at filter at ReadsSparkSource.java:96), which has no missing parents
17:38:19.440 INFO MemoryStore - Block broadcast_467 stored as values in memory (estimated size 148.2 KiB, free 1917.9 MiB)
17:38:19.440 INFO MemoryStore - Block broadcast_467_piece0 stored as bytes in memory (estimated size 54.6 KiB, free 1917.8 MiB)
17:38:19.440 INFO BlockManagerInfo - Added broadcast_467_piece0 in memory on localhost:37953 (size: 54.6 KiB, free: 1919.6 MiB)
17:38:19.440 INFO SparkContext - Created broadcast 467 from broadcast at DAGScheduler.scala:1580
17:38:19.441 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 232 (MapPartitionsRDD[1115] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:38:19.441 INFO TaskSchedulerImpl - Adding task set 232.0 with 1 tasks resource profile 0
17:38:19.441 INFO TaskSetManager - Starting task 0.0 in stage 232.0 (TID 288) (localhost, executor driver, partition 0, ANY, 7852 bytes)
17:38:19.441 INFO Executor - Running task 0.0 in stage 232.0 (TID 288)
17:38:19.453 INFO NewHadoopRDD - Input split: hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest1_b685c02d-74bf-4b11-88a9-99b561f1ae4a.bam:0+238969
17:38:19.454 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_b685c02d-74bf-4b11-88a9-99b561f1ae4a.bam dst=null perm=null proto=rpc
17:38:19.454 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_b685c02d-74bf-4b11-88a9-99b561f1ae4a.bam dst=null perm=null proto=rpc
17:38:19.455 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_b685c02d-74bf-4b11-88a9-99b561f1ae4a.bam.bai dst=null perm=null proto=rpc
17:38:19.456 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_b685c02d-74bf-4b11-88a9-99b561f1ae4a.bai dst=null perm=null proto=rpc
17:38:19.456 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:19.457 WARN DFSUtil - Unexpected value for data transfer bytes=5949 duration=0
17:38:19.460 WARN DFSUtil - Unexpected value for data transfer bytes=32 duration=0
17:38:19.462 INFO Executor - Finished task 0.0 in stage 232.0 (TID 288). 651483 bytes result sent to driver
17:38:19.463 INFO TaskSetManager - Finished task 0.0 in stage 232.0 (TID 288) in 22 ms on localhost (executor driver) (1/1)
17:38:19.463 INFO TaskSchedulerImpl - Removed TaskSet 232.0, whose tasks have all completed, from pool
17:38:19.464 INFO DAGScheduler - ResultStage 232 (collect at ReadsSparkSinkUnitTest.java:182) finished in 0.029 s
17:38:19.464 INFO DAGScheduler - Job 174 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:19.464 INFO TaskSchedulerImpl - Killing all running tasks in stage 232: Stage finished
17:38:19.464 INFO DAGScheduler - Job 174 finished: collect at ReadsSparkSinkUnitTest.java:182, took 0.030123 s
17:38:19.473 INFO SparkContext - Starting job: count at ReadsSparkSinkUnitTest.java:185
17:38:19.474 INFO DAGScheduler - Got job 175 (count at ReadsSparkSinkUnitTest.java:185) with 1 output partitions
17:38:19.474 INFO DAGScheduler - Final stage: ResultStage 233 (count at ReadsSparkSinkUnitTest.java:185)
17:38:19.474 INFO DAGScheduler - Parents of final stage: List()
17:38:19.474 INFO DAGScheduler - Missing parents: List()
17:38:19.474 INFO DAGScheduler - Submitting ResultStage 233 (MapPartitionsRDD[1097] at filter at ReadsSparkSource.java:96), which has no missing parents
17:38:19.489 INFO MemoryStore - Block broadcast_468 stored as values in memory (estimated size 426.1 KiB, free 1917.4 MiB)
17:38:19.493 INFO BlockManagerInfo - Removed broadcast_467_piece0 on localhost:37953 in memory (size: 54.6 KiB, free: 1919.7 MiB)
17:38:19.493 INFO BlockManagerInfo - Removed broadcast_462_piece0 on localhost:37953 in memory (size: 9.6 KiB, free: 1919.7 MiB)
17:38:19.493 INFO MemoryStore - Block broadcast_468_piece0 stored as bytes in memory (estimated size 153.6 KiB, free 1917.5 MiB)
17:38:19.493 INFO BlockManagerInfo - Added broadcast_468_piece0 in memory on localhost:37953 (size: 153.6 KiB, free: 1919.5 MiB)
17:38:19.494 INFO SparkContext - Created broadcast 468 from broadcast at DAGScheduler.scala:1580
17:38:19.494 INFO BlockManagerInfo - Removed broadcast_463_piece0 on localhost:37953 in memory (size: 166.1 KiB, free: 1919.7 MiB)
17:38:19.494 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 233 (MapPartitionsRDD[1097] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:38:19.494 INFO TaskSchedulerImpl - Adding task set 233.0 with 1 tasks resource profile 0
17:38:19.494 INFO TaskSetManager - Starting task 0.0 in stage 233.0 (TID 289) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7872 bytes)
17:38:19.495 INFO Executor - Running task 0.0 in stage 233.0 (TID 289)
17:38:19.495 INFO BlockManagerInfo - Removed broadcast_464_piece0 on localhost:37953 in memory (size: 67.1 KiB, free: 1919.7 MiB)
17:38:19.496 INFO BlockManagerInfo - Removed broadcast_461_piece0 on localhost:37953 in memory (size: 9.6 KiB, free: 1919.8 MiB)
17:38:19.525 INFO NewHadoopRDD - Input split: file:/home/runner/work/gatk/gatk/src/test/resources/org/broadinstitute/hellbender/tools/BQSR/HiSeq.1mb.1RG.2k_lines.bam:0+222075
17:38:19.526 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:19.527 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:19.534 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:19.537 INFO Executor - Finished task 0.0 in stage 233.0 (TID 289). 989 bytes result sent to driver
17:38:19.537 INFO TaskSetManager - Finished task 0.0 in stage 233.0 (TID 289) in 43 ms on localhost (executor driver) (1/1)
17:38:19.538 INFO TaskSchedulerImpl - Removed TaskSet 233.0, whose tasks have all completed, from pool
17:38:19.538 INFO DAGScheduler - ResultStage 233 (count at ReadsSparkSinkUnitTest.java:185) finished in 0.064 s
17:38:19.538 INFO DAGScheduler - Job 175 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:19.538 INFO TaskSchedulerImpl - Killing all running tasks in stage 233: Stage finished
17:38:19.538 INFO DAGScheduler - Job 175 finished: count at ReadsSparkSinkUnitTest.java:185, took 0.064477 s
17:38:19.541 INFO SparkContext - Starting job: count at ReadsSparkSinkUnitTest.java:185
17:38:19.541 INFO DAGScheduler - Got job 176 (count at ReadsSparkSinkUnitTest.java:185) with 1 output partitions
17:38:19.541 INFO DAGScheduler - Final stage: ResultStage 234 (count at ReadsSparkSinkUnitTest.java:185)
17:38:19.541 INFO DAGScheduler - Parents of final stage: List()
17:38:19.541 INFO DAGScheduler - Missing parents: List()
17:38:19.541 INFO DAGScheduler - Submitting ResultStage 234 (MapPartitionsRDD[1115] at filter at ReadsSparkSource.java:96), which has no missing parents
17:38:19.547 INFO MemoryStore - Block broadcast_469 stored as values in memory (estimated size 148.1 KiB, free 1918.6 MiB)
17:38:19.547 INFO MemoryStore - Block broadcast_469_piece0 stored as bytes in memory (estimated size 54.6 KiB, free 1918.6 MiB)
17:38:19.547 INFO BlockManagerInfo - Added broadcast_469_piece0 in memory on localhost:37953 (size: 54.6 KiB, free: 1919.7 MiB)
17:38:19.548 INFO SparkContext - Created broadcast 469 from broadcast at DAGScheduler.scala:1580
17:38:19.548 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 234 (MapPartitionsRDD[1115] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:38:19.548 INFO TaskSchedulerImpl - Adding task set 234.0 with 1 tasks resource profile 0
17:38:19.548 INFO TaskSetManager - Starting task 0.0 in stage 234.0 (TID 290) (localhost, executor driver, partition 0, ANY, 7852 bytes)
17:38:19.548 INFO Executor - Running task 0.0 in stage 234.0 (TID 290)
17:38:19.559 INFO NewHadoopRDD - Input split: hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest1_b685c02d-74bf-4b11-88a9-99b561f1ae4a.bam:0+238969
17:38:19.560 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_b685c02d-74bf-4b11-88a9-99b561f1ae4a.bam dst=null perm=null proto=rpc
17:38:19.560 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_b685c02d-74bf-4b11-88a9-99b561f1ae4a.bam dst=null perm=null proto=rpc
17:38:19.561 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_b685c02d-74bf-4b11-88a9-99b561f1ae4a.bam.bai dst=null perm=null proto=rpc
17:38:19.562 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_b685c02d-74bf-4b11-88a9-99b561f1ae4a.bai dst=null perm=null proto=rpc
17:38:19.562 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:19.563 WARN DFSUtil - Unexpected value for data transfer bytes=5949 duration=0
17:38:19.565 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:19.566 WARN DFSUtil - Unexpected value for data transfer bytes=32 duration=0
17:38:19.567 INFO Executor - Finished task 0.0 in stage 234.0 (TID 290). 989 bytes result sent to driver
17:38:19.568 INFO TaskSetManager - Finished task 0.0 in stage 234.0 (TID 290) in 20 ms on localhost (executor driver) (1/1)
17:38:19.568 INFO TaskSchedulerImpl - Removed TaskSet 234.0, whose tasks have all completed, from pool
17:38:19.568 INFO DAGScheduler - ResultStage 234 (count at ReadsSparkSinkUnitTest.java:185) finished in 0.027 s
17:38:19.568 INFO DAGScheduler - Job 176 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:19.568 INFO TaskSchedulerImpl - Killing all running tasks in stage 234: Stage finished
17:38:19.568 INFO DAGScheduler - Job 176 finished: count at ReadsSparkSinkUnitTest.java:185, took 0.027209 s
17:38:19.575 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_5e604a04-6a23-43e2-8662-3c2fcc2c3411.bam dst=null perm=null proto=rpc
17:38:19.576 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest1_5e604a04-6a23-43e2-8662-3c2fcc2c3411.bam dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:19.577 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest1_5e604a04-6a23-43e2-8662-3c2fcc2c3411.bam is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:38:19.578 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_5e604a04-6a23-43e2-8662-3c2fcc2c3411.bam dst=null perm=null proto=rpc
17:38:19.578 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:19.580 INFO MemoryStore - Block broadcast_470 stored as values in memory (estimated size 297.9 KiB, free 1918.3 MiB)
17:38:19.586 INFO MemoryStore - Block broadcast_470_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1918.2 MiB)
17:38:19.586 INFO BlockManagerInfo - Added broadcast_470_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.6 MiB)
17:38:19.586 INFO SparkContext - Created broadcast 470 from newAPIHadoopFile at PathSplitSource.java:96
17:38:19.605 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:19.607 INFO MemoryStore - Block broadcast_471 stored as values in memory (estimated size 297.9 KiB, free 1917.9 MiB)
17:38:19.612 INFO MemoryStore - Block broadcast_471_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1917.9 MiB)
17:38:19.612 INFO BlockManagerInfo - Added broadcast_471_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.6 MiB)
17:38:19.612 INFO SparkContext - Created broadcast 471 from newAPIHadoopFile at PathSplitSource.java:96
17:38:19.631 INFO FileInputFormat - Total input files to process : 1
17:38:19.633 INFO MemoryStore - Block broadcast_472 stored as values in memory (estimated size 160.7 KiB, free 1917.7 MiB)
17:38:19.633 INFO MemoryStore - Block broadcast_472_piece0 stored as bytes in memory (estimated size 9.6 KiB, free 1917.7 MiB)
17:38:19.633 INFO BlockManagerInfo - Added broadcast_472_piece0 in memory on localhost:37953 (size: 9.6 KiB, free: 1919.6 MiB)
17:38:19.634 INFO SparkContext - Created broadcast 472 from broadcast at ReadsSparkSink.java:133
17:38:19.634 WARN HtsjdkReadsRddStorage - Unrecognized write option: DISABLE
17:38:19.634 WARN HtsjdkReadsRddStorage - Unrecognized write option: DISABLE
17:38:19.635 INFO MemoryStore - Block broadcast_473 stored as values in memory (estimated size 163.2 KiB, free 1917.5 MiB)
17:38:19.635 INFO MemoryStore - Block broadcast_473_piece0 stored as bytes in memory (estimated size 9.6 KiB, free 1917.5 MiB)
17:38:19.635 INFO BlockManagerInfo - Added broadcast_473_piece0 in memory on localhost:37953 (size: 9.6 KiB, free: 1919.6 MiB)
17:38:19.636 INFO SparkContext - Created broadcast 473 from broadcast at BamSink.java:76
17:38:19.638 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_5e604a04-6a23-43e2-8662-3c2fcc2c3411.bam.parts dst=null perm=null proto=rpc
17:38:19.638 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:19.638 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:19.638 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:19.639 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=mkdirs src=/user/runner/ReadsSparkSinkUnitTest1_5e604a04-6a23-43e2-8662-3c2fcc2c3411.bam.parts/_temporary/0 dst=null perm=runner:supergroup:rwxr-xr-x proto=rpc
17:38:19.645 INFO SparkContext - Starting job: runJob at SparkHadoopWriter.scala:83
17:38:19.645 INFO DAGScheduler - Registering RDD 1129 (mapToPair at SparkUtils.java:161) as input to shuffle 47
17:38:19.645 INFO DAGScheduler - Got job 177 (runJob at SparkHadoopWriter.scala:83) with 1 output partitions
17:38:19.645 INFO DAGScheduler - Final stage: ResultStage 236 (runJob at SparkHadoopWriter.scala:83)
17:38:19.645 INFO DAGScheduler - Parents of final stage: List(ShuffleMapStage 235)
17:38:19.645 INFO DAGScheduler - Missing parents: List(ShuffleMapStage 235)
17:38:19.646 INFO DAGScheduler - Submitting ShuffleMapStage 235 (MapPartitionsRDD[1129] at mapToPair at SparkUtils.java:161), which has no missing parents
17:38:19.661 INFO MemoryStore - Block broadcast_474 stored as values in memory (estimated size 520.4 KiB, free 1917.0 MiB)
17:38:19.665 INFO BlockManagerInfo - Removed broadcast_459_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.6 MiB)
17:38:19.665 INFO BlockManagerInfo - Removed broadcast_469_piece0 on localhost:37953 in memory (size: 54.6 KiB, free: 1919.7 MiB)
17:38:19.665 INFO MemoryStore - Block broadcast_474_piece0 stored as bytes in memory (estimated size 166.1 KiB, free 1917.4 MiB)
17:38:19.665 INFO BlockManagerInfo - Added broadcast_474_piece0 in memory on localhost:37953 (size: 166.1 KiB, free: 1919.5 MiB)
17:38:19.665 INFO SparkContext - Created broadcast 474 from broadcast at DAGScheduler.scala:1580
17:38:19.665 INFO BlockManagerInfo - Removed broadcast_465_piece0 on localhost:37953 in memory (size: 233.0 B, free: 1919.5 MiB)
17:38:19.666 INFO DAGScheduler - Submitting 1 missing tasks from ShuffleMapStage 235 (MapPartitionsRDD[1129] at mapToPair at SparkUtils.java:161) (first 15 tasks are for partitions Vector(0))
17:38:19.666 INFO TaskSchedulerImpl - Adding task set 235.0 with 1 tasks resource profile 0
17:38:19.666 INFO BlockManagerInfo - Removed broadcast_471_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.6 MiB)
17:38:19.666 INFO TaskSetManager - Starting task 0.0 in stage 235.0 (TID 291) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7861 bytes)
17:38:19.666 INFO BlockManagerInfo - Removed broadcast_466_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.6 MiB)
17:38:19.666 INFO Executor - Running task 0.0 in stage 235.0 (TID 291)
17:38:19.667 INFO BlockManagerInfo - Removed broadcast_468_piece0 on localhost:37953 in memory (size: 153.6 KiB, free: 1919.8 MiB)
17:38:19.698 INFO NewHadoopRDD - Input split: file:/home/runner/work/gatk/gatk/src/test/resources/org/broadinstitute/hellbender/tools/BQSR/HiSeq.1mb.1RG.2k_lines.bam:0+222075
17:38:19.698 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:19.699 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:19.706 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:19.715 INFO Executor - Finished task 0.0 in stage 235.0 (TID 291). 1148 bytes result sent to driver
17:38:19.715 INFO TaskSetManager - Finished task 0.0 in stage 235.0 (TID 291) in 49 ms on localhost (executor driver) (1/1)
17:38:19.715 INFO TaskSchedulerImpl - Removed TaskSet 235.0, whose tasks have all completed, from pool
17:38:19.715 INFO DAGScheduler - ShuffleMapStage 235 (mapToPair at SparkUtils.java:161) finished in 0.069 s
17:38:19.715 INFO DAGScheduler - looking for newly runnable stages
17:38:19.715 INFO DAGScheduler - running: HashSet()
17:38:19.715 INFO DAGScheduler - waiting: HashSet(ResultStage 236)
17:38:19.715 INFO DAGScheduler - failed: HashSet()
17:38:19.715 INFO DAGScheduler - Submitting ResultStage 236 (MapPartitionsRDD[1134] at mapToPair at BamSink.java:91), which has no missing parents
17:38:19.721 INFO MemoryStore - Block broadcast_475 stored as values in memory (estimated size 241.5 KiB, free 1918.4 MiB)
17:38:19.722 INFO MemoryStore - Block broadcast_475_piece0 stored as bytes in memory (estimated size 67.1 KiB, free 1918.4 MiB)
17:38:19.722 INFO BlockManagerInfo - Added broadcast_475_piece0 in memory on localhost:37953 (size: 67.1 KiB, free: 1919.7 MiB)
17:38:19.722 INFO SparkContext - Created broadcast 475 from broadcast at DAGScheduler.scala:1580
17:38:19.722 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 236 (MapPartitionsRDD[1134] at mapToPair at BamSink.java:91) (first 15 tasks are for partitions Vector(0))
17:38:19.722 INFO TaskSchedulerImpl - Adding task set 236.0 with 1 tasks resource profile 0
17:38:19.723 INFO TaskSetManager - Starting task 0.0 in stage 236.0 (TID 292) (localhost, executor driver, partition 0, NODE_LOCAL, 7513 bytes)
17:38:19.723 INFO Executor - Running task 0.0 in stage 236.0 (TID 292)
17:38:19.728 INFO ShuffleBlockFetcherIterator - Getting 1 (343.8 KiB) non-empty blocks including 1 (343.8 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
17:38:19.728 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
17:38:19.739 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:19.739 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:19.739 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:19.739 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:19.739 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:19.739 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:19.740 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest1_5e604a04-6a23-43e2-8662-3c2fcc2c3411.bam.parts/_temporary/0/_temporary/attempt_202510271738194811950635134984368_1134_r_000000_0/part-r-00000 dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:19.741 WARN IntelDeflaterFactory - Intel Deflater not supported, using Java.util.zip.Deflater
17:38:19.744 INFO StateChange - BLOCK* allocate blk_1073741895_1071, replicas=127.0.0.1:37927 for /user/runner/ReadsSparkSinkUnitTest1_5e604a04-6a23-43e2-8662-3c2fcc2c3411.bam.parts/_temporary/0/_temporary/attempt_202510271738194811950635134984368_1134_r_000000_0/part-r-00000
17:38:19.745 INFO DataNode - Receiving BP-617366194-10.1.0.252-1761586664431:blk_1073741895_1071 src: /127.0.0.1:48926 dest: /127.0.0.1:37927
17:38:19.749 INFO clienttrace - src: /127.0.0.1:48926, dest: /127.0.0.1:37927, bytes: 233040, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1369393814_1, offset: 0, srvID: 379dd5e5-59c3-42cd-b9d3-67ca5e44560b, blockid: BP-617366194-10.1.0.252-1761586664431:blk_1073741895_1071, duration(ns): 3111674
17:38:19.749 INFO DataNode - PacketResponder: BP-617366194-10.1.0.252-1761586664431:blk_1073741895_1071, type=LAST_IN_PIPELINE terminating
17:38:19.750 INFO FSNamesystem - BLOCK* blk_1073741895_1071 is COMMITTED but not COMPLETE(numNodes= 0 < minimum = 1) in file /user/runner/ReadsSparkSinkUnitTest1_5e604a04-6a23-43e2-8662-3c2fcc2c3411.bam.parts/_temporary/0/_temporary/attempt_202510271738194811950635134984368_1134_r_000000_0/part-r-00000
17:38:20.150 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest1_5e604a04-6a23-43e2-8662-3c2fcc2c3411.bam.parts/_temporary/0/_temporary/attempt_202510271738194811950635134984368_1134_r_000000_0/part-r-00000 is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:38:20.151 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_5e604a04-6a23-43e2-8662-3c2fcc2c3411.bam.parts/_temporary/0/_temporary/attempt_202510271738194811950635134984368_1134_r_000000_0/part-r-00000 dst=null perm=null proto=rpc
17:38:20.152 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_5e604a04-6a23-43e2-8662-3c2fcc2c3411.bam.parts/_temporary/0/_temporary/attempt_202510271738194811950635134984368_1134_r_000000_0 dst=null perm=null proto=rpc
17:38:20.153 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_5e604a04-6a23-43e2-8662-3c2fcc2c3411.bam.parts/_temporary/0/_temporary/attempt_202510271738194811950635134984368_1134_r_000000_0 dst=null perm=null proto=rpc
17:38:20.153 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_5e604a04-6a23-43e2-8662-3c2fcc2c3411.bam.parts/_temporary/0/task_202510271738194811950635134984368_1134_r_000000 dst=null perm=null proto=rpc
17:38:20.154 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=rename src=/user/runner/ReadsSparkSinkUnitTest1_5e604a04-6a23-43e2-8662-3c2fcc2c3411.bam.parts/_temporary/0/_temporary/attempt_202510271738194811950635134984368_1134_r_000000_0 dst=/user/runner/ReadsSparkSinkUnitTest1_5e604a04-6a23-43e2-8662-3c2fcc2c3411.bam.parts/_temporary/0/task_202510271738194811950635134984368_1134_r_000000 perm=runner:supergroup:rwxr-xr-x proto=rpc
17:38:20.154 INFO FileOutputCommitter - Saved output of task 'attempt_202510271738194811950635134984368_1134_r_000000_0' to hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest1_5e604a04-6a23-43e2-8662-3c2fcc2c3411.bam.parts/_temporary/0/task_202510271738194811950635134984368_1134_r_000000
17:38:20.154 INFO SparkHadoopMapRedUtil - attempt_202510271738194811950635134984368_1134_r_000000_0: Committed. Elapsed time: 1 ms.
17:38:20.155 INFO Executor - Finished task 0.0 in stage 236.0 (TID 292). 1858 bytes result sent to driver
17:38:20.155 INFO TaskSetManager - Finished task 0.0 in stage 236.0 (TID 292) in 432 ms on localhost (executor driver) (1/1)
17:38:20.155 INFO TaskSchedulerImpl - Removed TaskSet 236.0, whose tasks have all completed, from pool
17:38:20.155 INFO DAGScheduler - ResultStage 236 (runJob at SparkHadoopWriter.scala:83) finished in 0.439 s
17:38:20.155 INFO DAGScheduler - Job 177 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:20.155 INFO TaskSchedulerImpl - Killing all running tasks in stage 236: Stage finished
17:38:20.155 INFO DAGScheduler - Job 177 finished: runJob at SparkHadoopWriter.scala:83, took 0.510643 s
17:38:20.156 INFO SparkHadoopWriter - Start to commit write Job job_202510271738194811950635134984368_1134.
17:38:20.156 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=listStatus src=/user/runner/ReadsSparkSinkUnitTest1_5e604a04-6a23-43e2-8662-3c2fcc2c3411.bam.parts/_temporary/0 dst=null perm=null proto=rpc
17:38:20.157 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_5e604a04-6a23-43e2-8662-3c2fcc2c3411.bam.parts dst=null perm=null proto=rpc
17:38:20.157 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=listStatus src=/user/runner/ReadsSparkSinkUnitTest1_5e604a04-6a23-43e2-8662-3c2fcc2c3411.bam.parts/_temporary/0/task_202510271738194811950635134984368_1134_r_000000 dst=null perm=null proto=rpc
17:38:20.158 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_5e604a04-6a23-43e2-8662-3c2fcc2c3411.bam.parts/part-r-00000 dst=null perm=null proto=rpc
17:38:20.159 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=rename src=/user/runner/ReadsSparkSinkUnitTest1_5e604a04-6a23-43e2-8662-3c2fcc2c3411.bam.parts/_temporary/0/task_202510271738194811950635134984368_1134_r_000000/part-r-00000 dst=/user/runner/ReadsSparkSinkUnitTest1_5e604a04-6a23-43e2-8662-3c2fcc2c3411.bam.parts/part-r-00000 perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:20.159 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=delete src=/user/runner/ReadsSparkSinkUnitTest1_5e604a04-6a23-43e2-8662-3c2fcc2c3411.bam.parts/_temporary dst=null perm=null proto=rpc
17:38:20.160 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest1_5e604a04-6a23-43e2-8662-3c2fcc2c3411.bam.parts/_SUCCESS dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:20.161 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest1_5e604a04-6a23-43e2-8662-3c2fcc2c3411.bam.parts/_SUCCESS is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:38:20.161 INFO audit - allowed=false ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=delete src=/user/runner/ReadsSparkSinkUnitTest1_5e604a04-6a23-43e2-8662-3c2fcc2c3411.bam.parts/.spark-staging-1134 dst=null perm=null proto=rpc
17:38:20.161 INFO SparkHadoopWriter - Write Job job_202510271738194811950635134984368_1134 committed. Elapsed time: 5 ms.
17:38:20.162 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest1_5e604a04-6a23-43e2-8662-3c2fcc2c3411.bam.parts/header dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:20.162 WARN IntelDeflaterFactory - Intel Deflater not supported, using Java.util.zip.Deflater
17:38:20.163 INFO StateChange - BLOCK* allocate blk_1073741896_1072, replicas=127.0.0.1:37927 for /user/runner/ReadsSparkSinkUnitTest1_5e604a04-6a23-43e2-8662-3c2fcc2c3411.bam.parts/header
17:38:20.164 INFO DataNode - Receiving BP-617366194-10.1.0.252-1761586664431:blk_1073741896_1072 src: /127.0.0.1:48936 dest: /127.0.0.1:37927
17:38:20.165 INFO clienttrace - src: /127.0.0.1:48936, dest: /127.0.0.1:37927, bytes: 5901, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1369393814_1, offset: 0, srvID: 379dd5e5-59c3-42cd-b9d3-67ca5e44560b, blockid: BP-617366194-10.1.0.252-1761586664431:blk_1073741896_1072, duration(ns): 512599
17:38:20.165 INFO DataNode - PacketResponder: BP-617366194-10.1.0.252-1761586664431:blk_1073741896_1072, type=LAST_IN_PIPELINE terminating
17:38:20.166 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest1_5e604a04-6a23-43e2-8662-3c2fcc2c3411.bam.parts/header is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:38:20.167 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest1_5e604a04-6a23-43e2-8662-3c2fcc2c3411.bam.parts/terminator dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:20.168 INFO StateChange - BLOCK* allocate blk_1073741897_1073, replicas=127.0.0.1:37927 for /user/runner/ReadsSparkSinkUnitTest1_5e604a04-6a23-43e2-8662-3c2fcc2c3411.bam.parts/terminator
17:38:20.168 INFO DataNode - Receiving BP-617366194-10.1.0.252-1761586664431:blk_1073741897_1073 src: /127.0.0.1:48944 dest: /127.0.0.1:37927
17:38:20.169 INFO clienttrace - src: /127.0.0.1:48944, dest: /127.0.0.1:37927, bytes: 28, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1369393814_1, offset: 0, srvID: 379dd5e5-59c3-42cd-b9d3-67ca5e44560b, blockid: BP-617366194-10.1.0.252-1761586664431:blk_1073741897_1073, duration(ns): 488742
17:38:20.169 INFO DataNode - PacketResponder: BP-617366194-10.1.0.252-1761586664431:blk_1073741897_1073, type=LAST_IN_PIPELINE terminating
17:38:20.170 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest1_5e604a04-6a23-43e2-8662-3c2fcc2c3411.bam.parts/terminator is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:38:20.170 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=listStatus src=/user/runner/ReadsSparkSinkUnitTest1_5e604a04-6a23-43e2-8662-3c2fcc2c3411.bam.parts dst=null perm=null proto=rpc
17:38:20.171 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest1_5e604a04-6a23-43e2-8662-3c2fcc2c3411.bam.parts/output dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:20.172 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest1_5e604a04-6a23-43e2-8662-3c2fcc2c3411.bam.parts/output is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:38:20.172 INFO HadoopFileSystemWrapper - Concatenating 3 parts to hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest1_5e604a04-6a23-43e2-8662-3c2fcc2c3411.bam
17:38:20.173 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=concat src=[/user/runner/ReadsSparkSinkUnitTest1_5e604a04-6a23-43e2-8662-3c2fcc2c3411.bam.parts/header, /user/runner/ReadsSparkSinkUnitTest1_5e604a04-6a23-43e2-8662-3c2fcc2c3411.bam.parts/part-r-00000, /user/runner/ReadsSparkSinkUnitTest1_5e604a04-6a23-43e2-8662-3c2fcc2c3411.bam.parts/terminator] dst=/user/runner/ReadsSparkSinkUnitTest1_5e604a04-6a23-43e2-8662-3c2fcc2c3411.bam.parts/output perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:20.173 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_5e604a04-6a23-43e2-8662-3c2fcc2c3411.bam dst=null perm=null proto=rpc
17:38:20.173 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=delete src=/user/runner/ReadsSparkSinkUnitTest1_5e604a04-6a23-43e2-8662-3c2fcc2c3411.bam dst=null perm=null proto=rpc
17:38:20.174 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=rename src=/user/runner/ReadsSparkSinkUnitTest1_5e604a04-6a23-43e2-8662-3c2fcc2c3411.bam.parts/output dst=/user/runner/ReadsSparkSinkUnitTest1_5e604a04-6a23-43e2-8662-3c2fcc2c3411.bam perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:20.174 INFO HadoopFileSystemWrapper - Concatenating to hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest1_5e604a04-6a23-43e2-8662-3c2fcc2c3411.bam done
17:38:20.174 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_5e604a04-6a23-43e2-8662-3c2fcc2c3411.bam dst=null perm=null proto=rpc
17:38:20.175 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=delete src=/user/runner/ReadsSparkSinkUnitTest1_5e604a04-6a23-43e2-8662-3c2fcc2c3411.bam.parts dst=null perm=null proto=rpc
17:38:20.175 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_5e604a04-6a23-43e2-8662-3c2fcc2c3411.bam dst=null perm=null proto=rpc
17:38:20.176 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_5e604a04-6a23-43e2-8662-3c2fcc2c3411.bam dst=null perm=null proto=rpc
17:38:20.177 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_5e604a04-6a23-43e2-8662-3c2fcc2c3411.bam dst=null perm=null proto=rpc
17:38:20.178 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_5e604a04-6a23-43e2-8662-3c2fcc2c3411.bam dst=null perm=null proto=rpc
17:38:20.178 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_5e604a04-6a23-43e2-8662-3c2fcc2c3411.bam.bai dst=null perm=null proto=rpc
17:38:20.179 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_5e604a04-6a23-43e2-8662-3c2fcc2c3411.bai dst=null perm=null proto=rpc
17:38:20.180 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:20.180 WARN DFSUtil - Unexpected value for data transfer bytes=5949 duration=0
17:38:20.182 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:20.182 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_5e604a04-6a23-43e2-8662-3c2fcc2c3411.bam.sbi dst=null perm=null proto=rpc
17:38:20.183 INFO MemoryStore - Block broadcast_476 stored as values in memory (estimated size 297.9 KiB, free 1918.1 MiB)
17:38:20.189 INFO MemoryStore - Block broadcast_476_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1918.0 MiB)
17:38:20.189 INFO BlockManagerInfo - Added broadcast_476_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.7 MiB)
17:38:20.189 INFO SparkContext - Created broadcast 476 from newAPIHadoopFile at PathSplitSource.java:96
17:38:20.209 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_5e604a04-6a23-43e2-8662-3c2fcc2c3411.bam dst=null perm=null proto=rpc
17:38:20.209 INFO FileInputFormat - Total input files to process : 1
17:38:20.209 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_5e604a04-6a23-43e2-8662-3c2fcc2c3411.bam dst=null perm=null proto=rpc
17:38:20.242 INFO SparkContext - Starting job: collect at ReadsSparkSinkUnitTest.java:182
17:38:20.242 INFO DAGScheduler - Got job 178 (collect at ReadsSparkSinkUnitTest.java:182) with 1 output partitions
17:38:20.242 INFO DAGScheduler - Final stage: ResultStage 237 (collect at ReadsSparkSinkUnitTest.java:182)
17:38:20.242 INFO DAGScheduler - Parents of final stage: List()
17:38:20.242 INFO DAGScheduler - Missing parents: List()
17:38:20.242 INFO DAGScheduler - Submitting ResultStage 237 (MapPartitionsRDD[1141] at filter at ReadsSparkSource.java:96), which has no missing parents
17:38:20.257 INFO MemoryStore - Block broadcast_477 stored as values in memory (estimated size 426.2 KiB, free 1917.6 MiB)
17:38:20.258 INFO MemoryStore - Block broadcast_477_piece0 stored as bytes in memory (estimated size 153.7 KiB, free 1917.4 MiB)
17:38:20.259 INFO BlockManagerInfo - Added broadcast_477_piece0 in memory on localhost:37953 (size: 153.7 KiB, free: 1919.5 MiB)
17:38:20.259 INFO SparkContext - Created broadcast 477 from broadcast at DAGScheduler.scala:1580
17:38:20.259 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 237 (MapPartitionsRDD[1141] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:38:20.259 INFO TaskSchedulerImpl - Adding task set 237.0 with 1 tasks resource profile 0
17:38:20.260 INFO TaskSetManager - Starting task 0.0 in stage 237.0 (TID 293) (localhost, executor driver, partition 0, ANY, 7852 bytes)
17:38:20.260 INFO Executor - Running task 0.0 in stage 237.0 (TID 293)
17:38:20.291 INFO NewHadoopRDD - Input split: hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest1_5e604a04-6a23-43e2-8662-3c2fcc2c3411.bam:0+238969
17:38:20.292 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_5e604a04-6a23-43e2-8662-3c2fcc2c3411.bam dst=null perm=null proto=rpc
17:38:20.292 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_5e604a04-6a23-43e2-8662-3c2fcc2c3411.bam dst=null perm=null proto=rpc
17:38:20.294 WARN DFSUtil - Unexpected value for data transfer bytes=5949 duration=0
17:38:20.294 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_5e604a04-6a23-43e2-8662-3c2fcc2c3411.bam dst=null perm=null proto=rpc
17:38:20.295 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_5e604a04-6a23-43e2-8662-3c2fcc2c3411.bam dst=null perm=null proto=rpc
17:38:20.295 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_5e604a04-6a23-43e2-8662-3c2fcc2c3411.bam.bai dst=null perm=null proto=rpc
17:38:20.296 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_5e604a04-6a23-43e2-8662-3c2fcc2c3411.bai dst=null perm=null proto=rpc
17:38:20.296 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:20.297 WARN DFSUtil - Unexpected value for data transfer bytes=5949 duration=0
17:38:20.299 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_5e604a04-6a23-43e2-8662-3c2fcc2c3411.bam dst=null perm=null proto=rpc
17:38:20.299 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_5e604a04-6a23-43e2-8662-3c2fcc2c3411.bam dst=null perm=null proto=rpc
17:38:20.299 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:20.300 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:20.300 WARN DFSUtil - Unexpected value for data transfer bytes=5949 duration=0
17:38:20.304 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:20.305 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:20.306 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:20.306 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:20.307 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:20.308 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:20.308 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:20.309 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:20.310 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:20.310 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:20.311 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:20.312 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:20.312 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:20.313 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:20.314 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:20.314 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:20.315 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:20.316 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:20.317 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:20.318 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:20.319 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:20.320 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:20.320 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:20.321 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:20.322 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:20.323 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:20.324 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:20.324 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:20.325 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:20.326 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:20.327 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:20.328 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:20.329 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:20.330 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:20.330 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:20.331 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:20.332 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:20.332 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:20.333 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:20.334 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:20.335 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:20.336 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:20.336 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:20.337 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:20.338 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:20.339 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:20.340 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:20.340 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:20.341 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:20.342 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:20.342 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:20.343 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:20.344 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:20.344 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:20.345 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:20.346 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:20.347 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:20.348 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:20.348 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:20.349 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:20.350 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:20.350 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_5e604a04-6a23-43e2-8662-3c2fcc2c3411.bam dst=null perm=null proto=rpc
17:38:20.351 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_5e604a04-6a23-43e2-8662-3c2fcc2c3411.bam dst=null perm=null proto=rpc
17:38:20.352 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_5e604a04-6a23-43e2-8662-3c2fcc2c3411.bam.bai dst=null perm=null proto=rpc
17:38:20.352 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_5e604a04-6a23-43e2-8662-3c2fcc2c3411.bai dst=null perm=null proto=rpc
17:38:20.353 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:20.354 WARN DFSUtil - Unexpected value for data transfer bytes=5949 duration=0
17:38:20.356 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:20.357 WARN DFSUtil - Unexpected value for data transfer bytes=32 duration=0
17:38:20.359 INFO Executor - Finished task 0.0 in stage 237.0 (TID 293). 651483 bytes result sent to driver
17:38:20.360 INFO TaskSetManager - Finished task 0.0 in stage 237.0 (TID 293) in 100 ms on localhost (executor driver) (1/1)
17:38:20.360 INFO TaskSchedulerImpl - Removed TaskSet 237.0, whose tasks have all completed, from pool
17:38:20.361 INFO DAGScheduler - ResultStage 237 (collect at ReadsSparkSinkUnitTest.java:182) finished in 0.118 s
17:38:20.361 INFO DAGScheduler - Job 178 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:20.361 INFO TaskSchedulerImpl - Killing all running tasks in stage 237: Stage finished
17:38:20.361 INFO DAGScheduler - Job 178 finished: collect at ReadsSparkSinkUnitTest.java:182, took 0.119108 s
17:38:20.371 INFO SparkContext - Starting job: count at ReadsSparkSinkUnitTest.java:185
17:38:20.371 INFO DAGScheduler - Got job 179 (count at ReadsSparkSinkUnitTest.java:185) with 1 output partitions
17:38:20.371 INFO DAGScheduler - Final stage: ResultStage 238 (count at ReadsSparkSinkUnitTest.java:185)
17:38:20.371 INFO DAGScheduler - Parents of final stage: List()
17:38:20.371 INFO DAGScheduler - Missing parents: List()
17:38:20.371 INFO DAGScheduler - Submitting ResultStage 238 (MapPartitionsRDD[1122] at filter at ReadsSparkSource.java:96), which has no missing parents
17:38:20.387 INFO MemoryStore - Block broadcast_478 stored as values in memory (estimated size 426.1 KiB, free 1917.0 MiB)
17:38:20.388 INFO MemoryStore - Block broadcast_478_piece0 stored as bytes in memory (estimated size 153.6 KiB, free 1916.9 MiB)
17:38:20.388 INFO BlockManagerInfo - Added broadcast_478_piece0 in memory on localhost:37953 (size: 153.6 KiB, free: 1919.4 MiB)
17:38:20.388 INFO SparkContext - Created broadcast 478 from broadcast at DAGScheduler.scala:1580
17:38:20.388 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 238 (MapPartitionsRDD[1122] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:38:20.388 INFO TaskSchedulerImpl - Adding task set 238.0 with 1 tasks resource profile 0
17:38:20.389 INFO TaskSetManager - Starting task 0.0 in stage 238.0 (TID 294) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7872 bytes)
17:38:20.389 INFO Executor - Running task 0.0 in stage 238.0 (TID 294)
17:38:20.420 INFO NewHadoopRDD - Input split: file:/home/runner/work/gatk/gatk/src/test/resources/org/broadinstitute/hellbender/tools/BQSR/HiSeq.1mb.1RG.2k_lines.bam:0+222075
17:38:20.420 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:20.421 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:20.428 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:20.432 INFO Executor - Finished task 0.0 in stage 238.0 (TID 294). 989 bytes result sent to driver
17:38:20.432 INFO TaskSetManager - Finished task 0.0 in stage 238.0 (TID 294) in 43 ms on localhost (executor driver) (1/1)
17:38:20.432 INFO TaskSchedulerImpl - Removed TaskSet 238.0, whose tasks have all completed, from pool
17:38:20.432 INFO DAGScheduler - ResultStage 238 (count at ReadsSparkSinkUnitTest.java:185) finished in 0.061 s
17:38:20.432 INFO DAGScheduler - Job 179 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:20.432 INFO TaskSchedulerImpl - Killing all running tasks in stage 238: Stage finished
17:38:20.433 INFO DAGScheduler - Job 179 finished: count at ReadsSparkSinkUnitTest.java:185, took 0.061776 s
17:38:20.436 INFO SparkContext - Starting job: count at ReadsSparkSinkUnitTest.java:185
17:38:20.436 INFO DAGScheduler - Got job 180 (count at ReadsSparkSinkUnitTest.java:185) with 1 output partitions
17:38:20.436 INFO DAGScheduler - Final stage: ResultStage 239 (count at ReadsSparkSinkUnitTest.java:185)
17:38:20.436 INFO DAGScheduler - Parents of final stage: List()
17:38:20.436 INFO DAGScheduler - Missing parents: List()
17:38:20.436 INFO DAGScheduler - Submitting ResultStage 239 (MapPartitionsRDD[1141] at filter at ReadsSparkSource.java:96), which has no missing parents
17:38:20.451 INFO MemoryStore - Block broadcast_479 stored as values in memory (estimated size 426.1 KiB, free 1916.5 MiB)
17:38:20.452 INFO MemoryStore - Block broadcast_479_piece0 stored as bytes in memory (estimated size 153.6 KiB, free 1916.3 MiB)
17:38:20.452 INFO BlockManagerInfo - Added broadcast_479_piece0 in memory on localhost:37953 (size: 153.6 KiB, free: 1919.2 MiB)
17:38:20.453 INFO SparkContext - Created broadcast 479 from broadcast at DAGScheduler.scala:1580
17:38:20.453 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 239 (MapPartitionsRDD[1141] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:38:20.453 INFO TaskSchedulerImpl - Adding task set 239.0 with 1 tasks resource profile 0
17:38:20.453 INFO TaskSetManager - Starting task 0.0 in stage 239.0 (TID 295) (localhost, executor driver, partition 0, ANY, 7852 bytes)
17:38:20.454 INFO Executor - Running task 0.0 in stage 239.0 (TID 295)
17:38:20.484 INFO NewHadoopRDD - Input split: hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest1_5e604a04-6a23-43e2-8662-3c2fcc2c3411.bam:0+238969
17:38:20.484 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_5e604a04-6a23-43e2-8662-3c2fcc2c3411.bam dst=null perm=null proto=rpc
17:38:20.485 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_5e604a04-6a23-43e2-8662-3c2fcc2c3411.bam dst=null perm=null proto=rpc
17:38:20.486 WARN DFSUtil - Unexpected value for data transfer bytes=5949 duration=0
17:38:20.486 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_5e604a04-6a23-43e2-8662-3c2fcc2c3411.bam dst=null perm=null proto=rpc
17:38:20.487 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_5e604a04-6a23-43e2-8662-3c2fcc2c3411.bam dst=null perm=null proto=rpc
17:38:20.488 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_5e604a04-6a23-43e2-8662-3c2fcc2c3411.bam.bai dst=null perm=null proto=rpc
17:38:20.488 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_5e604a04-6a23-43e2-8662-3c2fcc2c3411.bai dst=null perm=null proto=rpc
17:38:20.489 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:20.490 WARN DFSUtil - Unexpected value for data transfer bytes=5949 duration=0
17:38:20.491 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_5e604a04-6a23-43e2-8662-3c2fcc2c3411.bam dst=null perm=null proto=rpc
17:38:20.491 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_5e604a04-6a23-43e2-8662-3c2fcc2c3411.bam dst=null perm=null proto=rpc
17:38:20.492 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:20.492 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:20.493 WARN DFSUtil - Unexpected value for data transfer bytes=5949 duration=0
17:38:20.497 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:20.498 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:20.498 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:20.499 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:20.500 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:20.500 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:20.501 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:20.502 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:20.502 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:20.503 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:20.504 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:20.504 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:20.505 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:20.506 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:20.506 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:20.507 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:20.508 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:20.508 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:20.509 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:20.510 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:20.510 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:20.511 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:20.512 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:20.512 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:20.513 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:20.514 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:20.518 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:20.519 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:20.519 INFO BlockManagerInfo - Removed broadcast_472_piece0 on localhost:37953 in memory (size: 9.6 KiB, free: 1919.2 MiB)
17:38:20.520 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:20.520 INFO BlockManagerInfo - Removed broadcast_474_piece0 on localhost:37953 in memory (size: 166.1 KiB, free: 1919.4 MiB)
17:38:20.520 INFO BlockManagerInfo - Removed broadcast_475_piece0 on localhost:37953 in memory (size: 67.1 KiB, free: 1919.4 MiB)
17:38:20.521 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:20.521 INFO BlockManagerInfo - Removed broadcast_473_piece0 on localhost:37953 in memory (size: 9.6 KiB, free: 1919.5 MiB)
17:38:20.521 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:20.522 INFO BlockManagerInfo - Removed broadcast_477_piece0 on localhost:37953 in memory (size: 153.7 KiB, free: 1919.6 MiB)
17:38:20.522 INFO BlockManagerInfo - Removed broadcast_478_piece0 on localhost:37953 in memory (size: 153.6 KiB, free: 1919.8 MiB)
17:38:20.522 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:20.523 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:20.524 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:20.524 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:20.525 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:20.526 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:20.527 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:20.528 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:20.528 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:20.529 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:20.530 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:20.530 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:20.531 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:20.533 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:20.534 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:20.535 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:20.535 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:20.536 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:20.537 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:20.537 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:20.538 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:20.542 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:20.543 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:20.544 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:20.544 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:20.546 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:20.547 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:20.548 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:20.548 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:20.549 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:20.550 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:20.551 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:20.552 WARN DFSUtil - Unexpected value for data transfer bytes=1821 duration=0
17:38:20.552 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:20.553 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:20.553 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_5e604a04-6a23-43e2-8662-3c2fcc2c3411.bam dst=null perm=null proto=rpc
17:38:20.553 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest1_5e604a04-6a23-43e2-8662-3c2fcc2c3411.bam dst=null perm=null proto=rpc
17:38:20.554 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_5e604a04-6a23-43e2-8662-3c2fcc2c3411.bam.bai dst=null perm=null proto=rpc
17:38:20.555 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest1_5e604a04-6a23-43e2-8662-3c2fcc2c3411.bai dst=null perm=null proto=rpc
17:38:20.556 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:20.556 WARN DFSUtil - Unexpected value for data transfer bytes=5949 duration=0
17:38:20.559 WARN DFSUtil - Unexpected value for data transfer bytes=234864 duration=0
17:38:20.559 WARN DFSUtil - Unexpected value for data transfer bytes=32 duration=0
17:38:20.561 INFO Executor - Finished task 0.0 in stage 239.0 (TID 295). 1032 bytes result sent to driver
17:38:20.561 INFO TaskSetManager - Finished task 0.0 in stage 239.0 (TID 295) in 108 ms on localhost (executor driver) (1/1)
17:38:20.561 INFO TaskSchedulerImpl - Removed TaskSet 239.0, whose tasks have all completed, from pool
17:38:20.561 INFO DAGScheduler - ResultStage 239 (count at ReadsSparkSinkUnitTest.java:185) finished in 0.125 s
17:38:20.561 INFO DAGScheduler - Job 180 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:20.561 INFO TaskSchedulerImpl - Killing all running tasks in stage 239: Stage finished
17:38:20.561 INFO DAGScheduler - Job 180 finished: count at ReadsSparkSinkUnitTest.java:185, took 0.125801 s
17:38:20.570 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam dst=null perm=null proto=rpc
17:38:20.571 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:20.572 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:38:20.572 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam dst=null perm=null proto=rpc
17:38:20.573 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:20.575 INFO MemoryStore - Block broadcast_480 stored as values in memory (estimated size 298.0 KiB, free 1918.5 MiB)
17:38:20.580 INFO MemoryStore - Block broadcast_480_piece0 stored as bytes in memory (estimated size 50.3 KiB, free 1918.4 MiB)
17:38:20.580 INFO BlockManagerInfo - Added broadcast_480_piece0 in memory on localhost:37953 (size: 50.3 KiB, free: 1919.7 MiB)
17:38:20.581 INFO SparkContext - Created broadcast 480 from newAPIHadoopFile at PathSplitSource.java:96
17:38:20.599 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:20.601 INFO MemoryStore - Block broadcast_481 stored as values in memory (estimated size 298.0 KiB, free 1918.1 MiB)
17:38:20.607 INFO MemoryStore - Block broadcast_481_piece0 stored as bytes in memory (estimated size 50.3 KiB, free 1918.1 MiB)
17:38:20.607 INFO BlockManagerInfo - Added broadcast_481_piece0 in memory on localhost:37953 (size: 50.3 KiB, free: 1919.7 MiB)
17:38:20.607 INFO SparkContext - Created broadcast 481 from newAPIHadoopFile at PathSplitSource.java:96
17:38:20.626 INFO FileInputFormat - Total input files to process : 1
17:38:20.628 INFO MemoryStore - Block broadcast_482 stored as values in memory (estimated size 160.7 KiB, free 1917.9 MiB)
17:38:20.628 INFO MemoryStore - Block broadcast_482_piece0 stored as bytes in memory (estimated size 9.6 KiB, free 1917.9 MiB)
17:38:20.628 INFO BlockManagerInfo - Added broadcast_482_piece0 in memory on localhost:37953 (size: 9.6 KiB, free: 1919.6 MiB)
17:38:20.629 INFO SparkContext - Created broadcast 482 from broadcast at ReadsSparkSink.java:133
17:38:20.630 INFO MemoryStore - Block broadcast_483 stored as values in memory (estimated size 163.2 KiB, free 1917.7 MiB)
17:38:20.630 INFO MemoryStore - Block broadcast_483_piece0 stored as bytes in memory (estimated size 9.6 KiB, free 1917.7 MiB)
17:38:20.631 INFO BlockManagerInfo - Added broadcast_483_piece0 in memory on localhost:37953 (size: 9.6 KiB, free: 1919.6 MiB)
17:38:20.631 INFO SparkContext - Created broadcast 483 from broadcast at BamSink.java:76
17:38:20.633 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam.parts dst=null perm=null proto=rpc
17:38:20.633 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:20.634 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:20.634 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:20.634 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=mkdirs src=/user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam.parts/_temporary/0 dst=null perm=runner:supergroup:rwxr-xr-x proto=rpc
17:38:20.640 INFO SparkContext - Starting job: runJob at SparkHadoopWriter.scala:83
17:38:20.640 INFO DAGScheduler - Registering RDD 1155 (mapToPair at SparkUtils.java:161) as input to shuffle 48
17:38:20.640 INFO DAGScheduler - Got job 181 (runJob at SparkHadoopWriter.scala:83) with 1 output partitions
17:38:20.640 INFO DAGScheduler - Final stage: ResultStage 241 (runJob at SparkHadoopWriter.scala:83)
17:38:20.640 INFO DAGScheduler - Parents of final stage: List(ShuffleMapStage 240)
17:38:20.641 INFO DAGScheduler - Missing parents: List(ShuffleMapStage 240)
17:38:20.641 INFO DAGScheduler - Submitting ShuffleMapStage 240 (MapPartitionsRDD[1155] at mapToPair at SparkUtils.java:161), which has no missing parents
17:38:20.656 INFO MemoryStore - Block broadcast_484 stored as values in memory (estimated size 520.4 KiB, free 1917.2 MiB)
17:38:20.657 INFO MemoryStore - Block broadcast_484_piece0 stored as bytes in memory (estimated size 166.1 KiB, free 1917.1 MiB)
17:38:20.657 INFO BlockManagerInfo - Added broadcast_484_piece0 in memory on localhost:37953 (size: 166.1 KiB, free: 1919.5 MiB)
17:38:20.658 INFO SparkContext - Created broadcast 484 from broadcast at DAGScheduler.scala:1580
17:38:20.658 INFO DAGScheduler - Submitting 1 missing tasks from ShuffleMapStage 240 (MapPartitionsRDD[1155] at mapToPair at SparkUtils.java:161) (first 15 tasks are for partitions Vector(0))
17:38:20.658 INFO TaskSchedulerImpl - Adding task set 240.0 with 1 tasks resource profile 0
17:38:20.658 INFO TaskSetManager - Starting task 0.0 in stage 240.0 (TID 296) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7901 bytes)
17:38:20.659 INFO Executor - Running task 0.0 in stage 240.0 (TID 296)
17:38:20.690 INFO NewHadoopRDD - Input split: file:/home/runner/work/gatk/gatk/src/test/resources/org/broadinstitute/hellbender/tools/BQSR/expected.HiSeq.1mb.1RG.2k_lines.alternate.recalibrated.DIQ.bam:0+216896
17:38:20.691 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:20.692 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:20.700 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:20.708 INFO Executor - Finished task 0.0 in stage 240.0 (TID 296). 1148 bytes result sent to driver
17:38:20.709 INFO TaskSetManager - Finished task 0.0 in stage 240.0 (TID 296) in 51 ms on localhost (executor driver) (1/1)
17:38:20.709 INFO TaskSchedulerImpl - Removed TaskSet 240.0, whose tasks have all completed, from pool
17:38:20.709 INFO DAGScheduler - ShuffleMapStage 240 (mapToPair at SparkUtils.java:161) finished in 0.068 s
17:38:20.709 INFO DAGScheduler - looking for newly runnable stages
17:38:20.709 INFO DAGScheduler - running: HashSet()
17:38:20.709 INFO DAGScheduler - waiting: HashSet(ResultStage 241)
17:38:20.709 INFO DAGScheduler - failed: HashSet()
17:38:20.709 INFO DAGScheduler - Submitting ResultStage 241 (MapPartitionsRDD[1160] at mapToPair at BamSink.java:91), which has no missing parents
17:38:20.715 INFO MemoryStore - Block broadcast_485 stored as values in memory (estimated size 241.5 KiB, free 1916.8 MiB)
17:38:20.716 INFO MemoryStore - Block broadcast_485_piece0 stored as bytes in memory (estimated size 67.1 KiB, free 1916.8 MiB)
17:38:20.716 INFO BlockManagerInfo - Added broadcast_485_piece0 in memory on localhost:37953 (size: 67.1 KiB, free: 1919.4 MiB)
17:38:20.716 INFO SparkContext - Created broadcast 485 from broadcast at DAGScheduler.scala:1580
17:38:20.716 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 241 (MapPartitionsRDD[1160] at mapToPair at BamSink.java:91) (first 15 tasks are for partitions Vector(0))
17:38:20.716 INFO TaskSchedulerImpl - Adding task set 241.0 with 1 tasks resource profile 0
17:38:20.717 INFO TaskSetManager - Starting task 0.0 in stage 241.0 (TID 297) (localhost, executor driver, partition 0, NODE_LOCAL, 7513 bytes)
17:38:20.717 INFO Executor - Running task 0.0 in stage 241.0 (TID 297)
17:38:20.722 INFO ShuffleBlockFetcherIterator - Getting 1 (343.8 KiB) non-empty blocks including 1 (343.8 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
17:38:20.722 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
17:38:20.733 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:20.733 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:20.733 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:20.733 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:20.733 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:20.733 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:20.734 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam.parts/_temporary/0/_temporary/attempt_202510271738202792870646146493946_1160_r_000000_0/part-r-00000 dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:20.734 WARN IntelDeflaterFactory - Intel Deflater not supported, using Java.util.zip.Deflater
17:38:20.735 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam.parts/_temporary/0/_temporary/attempt_202510271738202792870646146493946_1160_r_000000_0/.part-r-00000.sbi dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:20.736 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam.parts/_temporary/0/_temporary/attempt_202510271738202792870646146493946_1160_r_000000_0/.part-r-00000.bai dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:20.742 INFO StateChange - BLOCK* allocate blk_1073741898_1074, replicas=127.0.0.1:37927 for /user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam.parts/_temporary/0/_temporary/attempt_202510271738202792870646146493946_1160_r_000000_0/part-r-00000
17:38:20.743 INFO DataNode - Receiving BP-617366194-10.1.0.252-1761586664431:blk_1073741898_1074 src: /127.0.0.1:49638 dest: /127.0.0.1:37927
17:38:20.745 INFO clienttrace - src: /127.0.0.1:49638, dest: /127.0.0.1:37927, bytes: 230392, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1369393814_1, offset: 0, srvID: 379dd5e5-59c3-42cd-b9d3-67ca5e44560b, blockid: BP-617366194-10.1.0.252-1761586664431:blk_1073741898_1074, duration(ns): 952588
17:38:20.745 INFO DataNode - PacketResponder: BP-617366194-10.1.0.252-1761586664431:blk_1073741898_1074, type=LAST_IN_PIPELINE terminating
17:38:20.745 INFO FSNamesystem - BLOCK* blk_1073741898_1074 is COMMITTED but not COMPLETE(numNodes= 0 < minimum = 1) in file /user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam.parts/_temporary/0/_temporary/attempt_202510271738202792870646146493946_1160_r_000000_0/part-r-00000
17:38:21.146 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam.parts/_temporary/0/_temporary/attempt_202510271738202792870646146493946_1160_r_000000_0/part-r-00000 is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:38:21.147 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam.parts/_temporary/0/_temporary/attempt_202510271738202792870646146493946_1160_r_000000_0/part-r-00000 dst=null perm=null proto=rpc
17:38:21.148 INFO StateChange - BLOCK* allocate blk_1073741899_1075, replicas=127.0.0.1:37927 for /user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam.parts/_temporary/0/_temporary/attempt_202510271738202792870646146493946_1160_r_000000_0/.part-r-00000.sbi
17:38:21.149 INFO DataNode - Receiving BP-617366194-10.1.0.252-1761586664431:blk_1073741899_1075 src: /127.0.0.1:49652 dest: /127.0.0.1:37927
17:38:21.150 INFO clienttrace - src: /127.0.0.1:49652, dest: /127.0.0.1:37927, bytes: 212, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1369393814_1, offset: 0, srvID: 379dd5e5-59c3-42cd-b9d3-67ca5e44560b, blockid: BP-617366194-10.1.0.252-1761586664431:blk_1073741899_1075, duration(ns): 475382
17:38:21.151 INFO DataNode - PacketResponder: BP-617366194-10.1.0.252-1761586664431:blk_1073741899_1075, type=LAST_IN_PIPELINE terminating
17:38:21.151 INFO FSNamesystem - BLOCK* blk_1073741899_1075 is COMMITTED but not COMPLETE(numNodes= 0 < minimum = 1) in file /user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam.parts/_temporary/0/_temporary/attempt_202510271738202792870646146493946_1160_r_000000_0/.part-r-00000.sbi
17:38:21.552 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam.parts/_temporary/0/_temporary/attempt_202510271738202792870646146493946_1160_r_000000_0/.part-r-00000.sbi is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:38:21.554 INFO StateChange - BLOCK* allocate blk_1073741900_1076, replicas=127.0.0.1:37927 for /user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam.parts/_temporary/0/_temporary/attempt_202510271738202792870646146493946_1160_r_000000_0/.part-r-00000.bai
17:38:21.555 INFO DataNode - Receiving BP-617366194-10.1.0.252-1761586664431:blk_1073741900_1076 src: /127.0.0.1:49660 dest: /127.0.0.1:37927
17:38:21.556 INFO clienttrace - src: /127.0.0.1:49660, dest: /127.0.0.1:37927, bytes: 5472, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1369393814_1, offset: 0, srvID: 379dd5e5-59c3-42cd-b9d3-67ca5e44560b, blockid: BP-617366194-10.1.0.252-1761586664431:blk_1073741900_1076, duration(ns): 576353
17:38:21.556 INFO DataNode - PacketResponder: BP-617366194-10.1.0.252-1761586664431:blk_1073741900_1076, type=LAST_IN_PIPELINE terminating
17:38:21.557 INFO FSNamesystem - BLOCK* blk_1073741900_1076 is COMMITTED but not COMPLETE(numNodes= 0 < minimum = 1) in file /user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam.parts/_temporary/0/_temporary/attempt_202510271738202792870646146493946_1160_r_000000_0/.part-r-00000.bai
17:38:21.600 INFO FsDatasetAsyncDiskService - Scheduling blk_1073741891_1067 replica FinalizedReplica, blk_1073741891_1067, FINALIZED
getNumBytes() = 212
getBytesOnDisk() = 212
getVisibleLength()= 212
getVolume() = /tmp/minicluster_storage14473397491642437130/data/data1
getBlockURI() = file:/tmp/minicluster_storage14473397491642437130/data/data1/current/BP-617366194-10.1.0.252-1761586664431/current/finalized/subdir0/subdir0/blk_1073741891 for deletion
17:38:21.600 INFO FsDatasetAsyncDiskService - Deleted BP-617366194-10.1.0.252-1761586664431 blk_1073741891_1067 URI file:/tmp/minicluster_storage14473397491642437130/data/data1/current/BP-617366194-10.1.0.252-1761586664431/current/finalized/subdir0/subdir0/blk_1073741891
17:38:21.958 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam.parts/_temporary/0/_temporary/attempt_202510271738202792870646146493946_1160_r_000000_0/.part-r-00000.bai is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:38:21.959 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam.parts/_temporary/0/_temporary/attempt_202510271738202792870646146493946_1160_r_000000_0 dst=null perm=null proto=rpc
17:38:21.960 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam.parts/_temporary/0/_temporary/attempt_202510271738202792870646146493946_1160_r_000000_0 dst=null perm=null proto=rpc
17:38:21.960 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam.parts/_temporary/0/task_202510271738202792870646146493946_1160_r_000000 dst=null perm=null proto=rpc
17:38:21.961 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=rename src=/user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam.parts/_temporary/0/_temporary/attempt_202510271738202792870646146493946_1160_r_000000_0 dst=/user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam.parts/_temporary/0/task_202510271738202792870646146493946_1160_r_000000 perm=runner:supergroup:rwxr-xr-x proto=rpc
17:38:21.961 INFO FileOutputCommitter - Saved output of task 'attempt_202510271738202792870646146493946_1160_r_000000_0' to hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam.parts/_temporary/0/task_202510271738202792870646146493946_1160_r_000000
17:38:21.961 INFO SparkHadoopMapRedUtil - attempt_202510271738202792870646146493946_1160_r_000000_0: Committed. Elapsed time: 1 ms.
17:38:21.961 INFO Executor - Finished task 0.0 in stage 241.0 (TID 297). 1858 bytes result sent to driver
17:38:21.962 INFO TaskSetManager - Finished task 0.0 in stage 241.0 (TID 297) in 1245 ms on localhost (executor driver) (1/1)
17:38:21.962 INFO TaskSchedulerImpl - Removed TaskSet 241.0, whose tasks have all completed, from pool
17:38:21.962 INFO DAGScheduler - ResultStage 241 (runJob at SparkHadoopWriter.scala:83) finished in 1.253 s
17:38:21.962 INFO DAGScheduler - Job 181 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:21.962 INFO TaskSchedulerImpl - Killing all running tasks in stage 241: Stage finished
17:38:21.962 INFO DAGScheduler - Job 181 finished: runJob at SparkHadoopWriter.scala:83, took 1.322377 s
17:38:21.963 INFO SparkHadoopWriter - Start to commit write Job job_202510271738202792870646146493946_1160.
17:38:21.963 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=listStatus src=/user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam.parts/_temporary/0 dst=null perm=null proto=rpc
17:38:21.964 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam.parts dst=null perm=null proto=rpc
17:38:21.964 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=listStatus src=/user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam.parts/_temporary/0/task_202510271738202792870646146493946_1160_r_000000 dst=null perm=null proto=rpc
17:38:21.964 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam.parts/.part-r-00000.bai dst=null perm=null proto=rpc
17:38:21.965 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=rename src=/user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam.parts/_temporary/0/task_202510271738202792870646146493946_1160_r_000000/.part-r-00000.bai dst=/user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam.parts/.part-r-00000.bai perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:21.965 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam.parts/.part-r-00000.sbi dst=null perm=null proto=rpc
17:38:21.966 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=rename src=/user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam.parts/_temporary/0/task_202510271738202792870646146493946_1160_r_000000/.part-r-00000.sbi dst=/user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam.parts/.part-r-00000.sbi perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:21.966 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam.parts/part-r-00000 dst=null perm=null proto=rpc
17:38:21.967 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=rename src=/user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam.parts/_temporary/0/task_202510271738202792870646146493946_1160_r_000000/part-r-00000 dst=/user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam.parts/part-r-00000 perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:21.968 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=delete src=/user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam.parts/_temporary dst=null perm=null proto=rpc
17:38:21.968 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam.parts/_SUCCESS dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:21.969 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam.parts/_SUCCESS is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:38:21.970 INFO audit - allowed=false ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=delete src=/user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam.parts/.spark-staging-1160 dst=null perm=null proto=rpc
17:38:21.970 INFO SparkHadoopWriter - Write Job job_202510271738202792870646146493946_1160 committed. Elapsed time: 7 ms.
17:38:21.970 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam.parts/header dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:21.971 WARN IntelDeflaterFactory - Intel Deflater not supported, using Java.util.zip.Deflater
17:38:21.972 INFO StateChange - BLOCK* allocate blk_1073741901_1077, replicas=127.0.0.1:37927 for /user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam.parts/header
17:38:21.973 INFO DataNode - Receiving BP-617366194-10.1.0.252-1761586664431:blk_1073741901_1077 src: /127.0.0.1:49666 dest: /127.0.0.1:37927
17:38:21.974 INFO clienttrace - src: /127.0.0.1:49666, dest: /127.0.0.1:37927, bytes: 5901, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1369393814_1, offset: 0, srvID: 379dd5e5-59c3-42cd-b9d3-67ca5e44560b, blockid: BP-617366194-10.1.0.252-1761586664431:blk_1073741901_1077, duration(ns): 571657
17:38:21.974 INFO DataNode - PacketResponder: BP-617366194-10.1.0.252-1761586664431:blk_1073741901_1077, type=LAST_IN_PIPELINE terminating
17:38:21.974 INFO FSNamesystem - BLOCK* blk_1073741901_1077 is COMMITTED but not COMPLETE(numNodes= 0 < minimum = 1) in file /user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam.parts/header
17:38:22.376 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam.parts/header is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:38:22.377 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam.parts/terminator dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:22.378 INFO StateChange - BLOCK* allocate blk_1073741902_1078, replicas=127.0.0.1:37927 for /user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam.parts/terminator
17:38:22.379 INFO DataNode - Receiving BP-617366194-10.1.0.252-1761586664431:blk_1073741902_1078 src: /127.0.0.1:49674 dest: /127.0.0.1:37927
17:38:22.380 INFO clienttrace - src: /127.0.0.1:49674, dest: /127.0.0.1:37927, bytes: 28, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1369393814_1, offset: 0, srvID: 379dd5e5-59c3-42cd-b9d3-67ca5e44560b, blockid: BP-617366194-10.1.0.252-1761586664431:blk_1073741902_1078, duration(ns): 526871
17:38:22.380 INFO DataNode - PacketResponder: BP-617366194-10.1.0.252-1761586664431:blk_1073741902_1078, type=LAST_IN_PIPELINE terminating
17:38:22.381 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam.parts/terminator is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:38:22.381 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=listStatus src=/user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam.parts dst=null perm=null proto=rpc
17:38:22.382 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam.parts/output dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:22.383 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam.parts/output is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:38:22.383 INFO HadoopFileSystemWrapper - Concatenating 3 parts to hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam
17:38:22.384 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=concat src=[/user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam.parts/header, /user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam.parts/part-r-00000, /user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam.parts/terminator] dst=/user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam.parts/output perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:22.384 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam dst=null perm=null proto=rpc
17:38:22.385 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=delete src=/user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam dst=null perm=null proto=rpc
17:38:22.385 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=rename src=/user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam.parts/output dst=/user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:22.385 INFO HadoopFileSystemWrapper - Concatenating to hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam done
17:38:22.386 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam dst=null perm=null proto=rpc
17:38:22.386 INFO IndexFileMerger - Merging .sbi files in temp directory hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam.parts/ to hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam.sbi
17:38:22.386 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=listStatus src=/user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam.parts dst=null perm=null proto=rpc
17:38:22.387 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam.sbi dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:22.388 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam.parts/.part-r-00000.sbi dst=null perm=null proto=rpc
17:38:22.388 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam.parts/.part-r-00000.sbi dst=null perm=null proto=rpc
17:38:22.389 WARN DFSUtil - Unexpected value for data transfer bytes=216 duration=0
17:38:22.390 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=delete src=/user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam.parts/.part-r-00000.sbi dst=null perm=null proto=rpc
17:38:22.391 INFO StateChange - BLOCK* allocate blk_1073741903_1079, replicas=127.0.0.1:37927 for /user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam.sbi
17:38:22.391 INFO DataNode - Receiving BP-617366194-10.1.0.252-1761586664431:blk_1073741903_1079 src: /127.0.0.1:49686 dest: /127.0.0.1:37927
17:38:22.392 INFO clienttrace - src: /127.0.0.1:49686, dest: /127.0.0.1:37927, bytes: 212, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1369393814_1, offset: 0, srvID: 379dd5e5-59c3-42cd-b9d3-67ca5e44560b, blockid: BP-617366194-10.1.0.252-1761586664431:blk_1073741903_1079, duration(ns): 455142
17:38:22.392 INFO DataNode - PacketResponder: BP-617366194-10.1.0.252-1761586664431:blk_1073741903_1079, type=LAST_IN_PIPELINE terminating
17:38:22.393 INFO FSNamesystem - BLOCK* blk_1073741903_1079 is COMMITTED but not COMPLETE(numNodes= 0 < minimum = 1) in file /user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam.sbi
17:38:22.794 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam.sbi is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:38:22.794 INFO IndexFileMerger - Done merging .sbi files
17:38:22.795 INFO IndexFileMerger - Merging .bai files in temp directory hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam.parts/ to hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam.bai
17:38:22.795 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=listStatus src=/user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam.parts dst=null perm=null proto=rpc
17:38:22.796 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam.bai dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:22.797 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam.parts/.part-r-00000.bai dst=null perm=null proto=rpc
17:38:22.797 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam.parts/.part-r-00000.bai dst=null perm=null proto=rpc
17:38:22.798 WARN DFSUtil - Unexpected value for data transfer bytes=5516 duration=0
17:38:22.799 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=delete src=/user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam.parts/.part-r-00000.bai dst=null perm=null proto=rpc
17:38:22.800 INFO StateChange - BLOCK* allocate blk_1073741904_1080, replicas=127.0.0.1:37927 for /user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam.bai
17:38:22.801 INFO DataNode - Receiving BP-617366194-10.1.0.252-1761586664431:blk_1073741904_1080 src: /127.0.0.1:49688 dest: /127.0.0.1:37927
17:38:22.802 INFO clienttrace - src: /127.0.0.1:49688, dest: /127.0.0.1:37927, bytes: 5472, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1369393814_1, offset: 0, srvID: 379dd5e5-59c3-42cd-b9d3-67ca5e44560b, blockid: BP-617366194-10.1.0.252-1761586664431:blk_1073741904_1080, duration(ns): 506743
17:38:22.802 INFO DataNode - PacketResponder: BP-617366194-10.1.0.252-1761586664431:blk_1073741904_1080, type=LAST_IN_PIPELINE terminating
17:38:22.803 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam.bai is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:38:22.803 INFO IndexFileMerger - Done merging .bai files
17:38:22.803 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=delete src=/user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam.parts dst=null perm=null proto=rpc
17:38:22.812 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam.bai dst=null perm=null proto=rpc
17:38:22.819 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam.sbi dst=null perm=null proto=rpc
17:38:22.820 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam.sbi dst=null perm=null proto=rpc
17:38:22.820 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam.sbi dst=null perm=null proto=rpc
17:38:22.821 WARN DFSUtil - Unexpected value for data transfer bytes=216 duration=0
17:38:22.822 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam dst=null perm=null proto=rpc
17:38:22.822 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam dst=null perm=null proto=rpc
17:38:22.823 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam dst=null perm=null proto=rpc
17:38:22.823 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam dst=null perm=null proto=rpc
17:38:22.824 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam.bai dst=null perm=null proto=rpc
17:38:22.824 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam.bai dst=null perm=null proto=rpc
17:38:22.825 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam.bai dst=null perm=null proto=rpc
17:38:22.826 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:22.826 WARN DFSUtil - Unexpected value for data transfer bytes=5949 duration=0
17:38:22.828 WARN DFSUtil - Unexpected value for data transfer bytes=5516 duration=0
17:38:22.828 WARN DFSUtil - Unexpected value for data transfer bytes=5516 duration=0
17:38:22.829 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam.sbi dst=null perm=null proto=rpc
17:38:22.829 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam.sbi dst=null perm=null proto=rpc
17:38:22.830 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam.sbi dst=null perm=null proto=rpc
17:38:22.830 WARN DFSUtil - Unexpected value for data transfer bytes=216 duration=0
17:38:22.831 INFO MemoryStore - Block broadcast_486 stored as values in memory (estimated size 320.0 B, free 1916.8 MiB)
17:38:22.834 INFO MemoryStore - Block broadcast_486_piece0 stored as bytes in memory (estimated size 233.0 B, free 1916.8 MiB)
17:38:22.834 INFO BlockManagerInfo - Added broadcast_486_piece0 in memory on localhost:37953 (size: 233.0 B, free: 1919.4 MiB)
17:38:22.834 INFO BlockManagerInfo - Removed broadcast_482_piece0 on localhost:37953 in memory (size: 9.6 KiB, free: 1919.4 MiB)
17:38:22.835 INFO SparkContext - Created broadcast 486 from broadcast at BamSource.java:104
17:38:22.835 INFO BlockManagerInfo - Removed broadcast_484_piece0 on localhost:37953 in memory (size: 166.1 KiB, free: 1919.6 MiB)
17:38:22.835 INFO BlockManagerInfo - Removed broadcast_470_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.6 MiB)
17:38:22.836 INFO MemoryStore - Block broadcast_487 stored as values in memory (estimated size 297.9 KiB, free 1917.7 MiB)
17:38:22.836 INFO BlockManagerInfo - Removed broadcast_476_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.7 MiB)
17:38:22.836 INFO BlockManagerInfo - Removed broadcast_485_piece0 on localhost:37953 in memory (size: 67.1 KiB, free: 1919.7 MiB)
17:38:22.837 INFO BlockManagerInfo - Removed broadcast_479_piece0 on localhost:37953 in memory (size: 153.6 KiB, free: 1919.9 MiB)
17:38:22.837 INFO BlockManagerInfo - Removed broadcast_481_piece0 on localhost:37953 in memory (size: 50.3 KiB, free: 1919.9 MiB)
17:38:22.838 INFO BlockManagerInfo - Removed broadcast_483_piece0 on localhost:37953 in memory (size: 9.6 KiB, free: 1920.0 MiB)
17:38:22.841 INFO MemoryStore - Block broadcast_487_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1919.3 MiB)
17:38:22.841 INFO BlockManagerInfo - Added broadcast_487_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.9 MiB)
17:38:22.841 INFO SparkContext - Created broadcast 487 from newAPIHadoopFile at PathSplitSource.java:96
17:38:22.850 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam dst=null perm=null proto=rpc
17:38:22.851 INFO FileInputFormat - Total input files to process : 1
17:38:22.851 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam dst=null perm=null proto=rpc
17:38:22.864 INFO SparkContext - Starting job: collect at ReadsSparkSinkUnitTest.java:182
17:38:22.865 INFO DAGScheduler - Got job 182 (collect at ReadsSparkSinkUnitTest.java:182) with 1 output partitions
17:38:22.865 INFO DAGScheduler - Final stage: ResultStage 242 (collect at ReadsSparkSinkUnitTest.java:182)
17:38:22.865 INFO DAGScheduler - Parents of final stage: List()
17:38:22.865 INFO DAGScheduler - Missing parents: List()
17:38:22.865 INFO DAGScheduler - Submitting ResultStage 242 (MapPartitionsRDD[1166] at filter at ReadsSparkSource.java:96), which has no missing parents
17:38:22.871 INFO MemoryStore - Block broadcast_488 stored as values in memory (estimated size 148.2 KiB, free 1919.2 MiB)
17:38:22.871 INFO MemoryStore - Block broadcast_488_piece0 stored as bytes in memory (estimated size 54.6 KiB, free 1919.1 MiB)
17:38:22.871 INFO BlockManagerInfo - Added broadcast_488_piece0 in memory on localhost:37953 (size: 54.6 KiB, free: 1919.8 MiB)
17:38:22.871 INFO SparkContext - Created broadcast 488 from broadcast at DAGScheduler.scala:1580
17:38:22.872 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 242 (MapPartitionsRDD[1166] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:38:22.872 INFO TaskSchedulerImpl - Adding task set 242.0 with 1 tasks resource profile 0
17:38:22.872 INFO TaskSetManager - Starting task 0.0 in stage 242.0 (TID 298) (localhost, executor driver, partition 0, ANY, 7852 bytes)
17:38:22.873 INFO Executor - Running task 0.0 in stage 242.0 (TID 298)
17:38:22.885 INFO NewHadoopRDD - Input split: hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam:0+236321
17:38:22.886 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam dst=null perm=null proto=rpc
17:38:22.886 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam dst=null perm=null proto=rpc
17:38:22.887 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam.bai dst=null perm=null proto=rpc
17:38:22.887 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam.bai dst=null perm=null proto=rpc
17:38:22.888 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam.bai dst=null perm=null proto=rpc
17:38:22.889 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:22.889 WARN DFSUtil - Unexpected value for data transfer bytes=5949 duration=0
17:38:22.891 WARN DFSUtil - Unexpected value for data transfer bytes=5516 duration=0
17:38:22.892 WARN DFSUtil - Unexpected value for data transfer bytes=5516 duration=0
17:38:22.893 WARN DFSUtil - Unexpected value for data transfer bytes=232192 duration=0
17:38:22.893 WARN DFSUtil - Unexpected value for data transfer bytes=32 duration=0
17:38:22.896 INFO Executor - Finished task 0.0 in stage 242.0 (TID 298). 650141 bytes result sent to driver
17:38:22.897 INFO TaskSetManager - Finished task 0.0 in stage 242.0 (TID 298) in 25 ms on localhost (executor driver) (1/1)
17:38:22.897 INFO TaskSchedulerImpl - Removed TaskSet 242.0, whose tasks have all completed, from pool
17:38:22.897 INFO DAGScheduler - ResultStage 242 (collect at ReadsSparkSinkUnitTest.java:182) finished in 0.032 s
17:38:22.897 INFO DAGScheduler - Job 182 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:22.897 INFO TaskSchedulerImpl - Killing all running tasks in stage 242: Stage finished
17:38:22.897 INFO DAGScheduler - Job 182 finished: collect at ReadsSparkSinkUnitTest.java:182, took 0.032933 s
17:38:22.907 INFO SparkContext - Starting job: count at ReadsSparkSinkUnitTest.java:185
17:38:22.907 INFO DAGScheduler - Got job 183 (count at ReadsSparkSinkUnitTest.java:185) with 1 output partitions
17:38:22.907 INFO DAGScheduler - Final stage: ResultStage 243 (count at ReadsSparkSinkUnitTest.java:185)
17:38:22.907 INFO DAGScheduler - Parents of final stage: List()
17:38:22.907 INFO DAGScheduler - Missing parents: List()
17:38:22.908 INFO DAGScheduler - Submitting ResultStage 243 (MapPartitionsRDD[1148] at filter at ReadsSparkSource.java:96), which has no missing parents
17:38:22.923 INFO MemoryStore - Block broadcast_489 stored as values in memory (estimated size 426.1 KiB, free 1918.7 MiB)
17:38:22.924 INFO MemoryStore - Block broadcast_489_piece0 stored as bytes in memory (estimated size 153.6 KiB, free 1918.6 MiB)
17:38:22.924 INFO BlockManagerInfo - Added broadcast_489_piece0 in memory on localhost:37953 (size: 153.6 KiB, free: 1919.7 MiB)
17:38:22.924 INFO SparkContext - Created broadcast 489 from broadcast at DAGScheduler.scala:1580
17:38:22.924 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 243 (MapPartitionsRDD[1148] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:38:22.924 INFO TaskSchedulerImpl - Adding task set 243.0 with 1 tasks resource profile 0
17:38:22.925 INFO TaskSetManager - Starting task 0.0 in stage 243.0 (TID 299) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7912 bytes)
17:38:22.925 INFO Executor - Running task 0.0 in stage 243.0 (TID 299)
17:38:22.956 INFO NewHadoopRDD - Input split: file:/home/runner/work/gatk/gatk/src/test/resources/org/broadinstitute/hellbender/tools/BQSR/expected.HiSeq.1mb.1RG.2k_lines.alternate.recalibrated.DIQ.bam:0+216896
17:38:22.957 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:22.958 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:22.967 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:22.970 INFO Executor - Finished task 0.0 in stage 243.0 (TID 299). 989 bytes result sent to driver
17:38:22.971 INFO TaskSetManager - Finished task 0.0 in stage 243.0 (TID 299) in 46 ms on localhost (executor driver) (1/1)
17:38:22.971 INFO TaskSchedulerImpl - Removed TaskSet 243.0, whose tasks have all completed, from pool
17:38:22.971 INFO DAGScheduler - ResultStage 243 (count at ReadsSparkSinkUnitTest.java:185) finished in 0.063 s
17:38:22.971 INFO DAGScheduler - Job 183 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:22.971 INFO TaskSchedulerImpl - Killing all running tasks in stage 243: Stage finished
17:38:22.971 INFO DAGScheduler - Job 183 finished: count at ReadsSparkSinkUnitTest.java:185, took 0.063783 s
17:38:22.974 INFO SparkContext - Starting job: count at ReadsSparkSinkUnitTest.java:185
17:38:22.974 INFO DAGScheduler - Got job 184 (count at ReadsSparkSinkUnitTest.java:185) with 1 output partitions
17:38:22.974 INFO DAGScheduler - Final stage: ResultStage 244 (count at ReadsSparkSinkUnitTest.java:185)
17:38:22.974 INFO DAGScheduler - Parents of final stage: List()
17:38:22.974 INFO DAGScheduler - Missing parents: List()
17:38:22.975 INFO DAGScheduler - Submitting ResultStage 244 (MapPartitionsRDD[1166] at filter at ReadsSparkSource.java:96), which has no missing parents
17:38:22.980 INFO MemoryStore - Block broadcast_490 stored as values in memory (estimated size 148.1 KiB, free 1918.4 MiB)
17:38:22.981 INFO MemoryStore - Block broadcast_490_piece0 stored as bytes in memory (estimated size 54.6 KiB, free 1918.4 MiB)
17:38:22.981 INFO BlockManagerInfo - Added broadcast_490_piece0 in memory on localhost:37953 (size: 54.6 KiB, free: 1919.6 MiB)
17:38:22.981 INFO SparkContext - Created broadcast 490 from broadcast at DAGScheduler.scala:1580
17:38:22.981 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 244 (MapPartitionsRDD[1166] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:38:22.981 INFO TaskSchedulerImpl - Adding task set 244.0 with 1 tasks resource profile 0
17:38:22.982 INFO TaskSetManager - Starting task 0.0 in stage 244.0 (TID 300) (localhost, executor driver, partition 0, ANY, 7852 bytes)
17:38:22.982 INFO Executor - Running task 0.0 in stage 244.0 (TID 300)
17:38:22.994 INFO NewHadoopRDD - Input split: hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam:0+236321
17:38:22.994 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam dst=null perm=null proto=rpc
17:38:22.995 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam dst=null perm=null proto=rpc
17:38:22.996 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam.bai dst=null perm=null proto=rpc
17:38:22.996 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam.bai dst=null perm=null proto=rpc
17:38:22.997 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest2_646c5a51-278f-41e4-ae94-5d7796516c3b.bam.bai dst=null perm=null proto=rpc
17:38:22.998 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:22.998 WARN DFSUtil - Unexpected value for data transfer bytes=5949 duration=0
17:38:23.000 WARN DFSUtil - Unexpected value for data transfer bytes=5516 duration=0
17:38:23.000 WARN DFSUtil - Unexpected value for data transfer bytes=5516 duration=0
17:38:23.001 WARN DFSUtil - Unexpected value for data transfer bytes=232192 duration=0
17:38:23.002 WARN DFSUtil - Unexpected value for data transfer bytes=32 duration=0
17:38:23.003 INFO Executor - Finished task 0.0 in stage 244.0 (TID 300). 989 bytes result sent to driver
17:38:23.004 INFO TaskSetManager - Finished task 0.0 in stage 244.0 (TID 300) in 22 ms on localhost (executor driver) (1/1)
17:38:23.004 INFO TaskSchedulerImpl - Removed TaskSet 244.0, whose tasks have all completed, from pool
17:38:23.004 INFO DAGScheduler - ResultStage 244 (count at ReadsSparkSinkUnitTest.java:185) finished in 0.029 s
17:38:23.004 INFO DAGScheduler - Job 184 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:23.004 INFO TaskSchedulerImpl - Killing all running tasks in stage 244: Stage finished
17:38:23.004 INFO DAGScheduler - Job 184 finished: count at ReadsSparkSinkUnitTest.java:185, took 0.029793 s
17:38:23.012 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam dst=null perm=null proto=rpc
17:38:23.013 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:23.014 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:38:23.014 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam dst=null perm=null proto=rpc
17:38:23.015 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:23.016 INFO MemoryStore - Block broadcast_491 stored as values in memory (estimated size 298.0 KiB, free 1918.1 MiB)
17:38:23.022 INFO MemoryStore - Block broadcast_491_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1918.0 MiB)
17:38:23.022 INFO BlockManagerInfo - Added broadcast_491_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.6 MiB)
17:38:23.022 INFO SparkContext - Created broadcast 491 from newAPIHadoopFile at PathSplitSource.java:96
17:38:23.041 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:23.043 INFO MemoryStore - Block broadcast_492 stored as values in memory (estimated size 298.0 KiB, free 1917.7 MiB)
17:38:23.047 INFO BlockManagerInfo - Removed broadcast_489_piece0 on localhost:37953 in memory (size: 153.6 KiB, free: 1919.7 MiB)
17:38:23.048 INFO BlockManagerInfo - Removed broadcast_480_piece0 on localhost:37953 in memory (size: 50.3 KiB, free: 1919.8 MiB)
17:38:23.048 INFO BlockManagerInfo - Removed broadcast_486_piece0 on localhost:37953 in memory (size: 233.0 B, free: 1919.8 MiB)
17:38:23.049 INFO BlockManagerInfo - Removed broadcast_487_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.8 MiB)
17:38:23.049 INFO BlockManagerInfo - Removed broadcast_488_piece0 on localhost:37953 in memory (size: 54.6 KiB, free: 1919.9 MiB)
17:38:23.049 INFO BlockManagerInfo - Removed broadcast_490_piece0 on localhost:37953 in memory (size: 54.6 KiB, free: 1920.0 MiB)
17:38:23.051 INFO MemoryStore - Block broadcast_492_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1919.3 MiB)
17:38:23.051 INFO BlockManagerInfo - Added broadcast_492_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.9 MiB)
17:38:23.051 INFO SparkContext - Created broadcast 492 from newAPIHadoopFile at PathSplitSource.java:96
17:38:23.070 INFO FileInputFormat - Total input files to process : 1
17:38:23.071 INFO MemoryStore - Block broadcast_493 stored as values in memory (estimated size 19.6 KiB, free 1919.3 MiB)
17:38:23.072 INFO MemoryStore - Block broadcast_493_piece0 stored as bytes in memory (estimated size 1890.0 B, free 1919.3 MiB)
17:38:23.072 INFO BlockManagerInfo - Added broadcast_493_piece0 in memory on localhost:37953 (size: 1890.0 B, free: 1919.9 MiB)
17:38:23.072 INFO SparkContext - Created broadcast 493 from broadcast at ReadsSparkSink.java:133
17:38:23.073 INFO MemoryStore - Block broadcast_494 stored as values in memory (estimated size 20.0 KiB, free 1919.3 MiB)
17:38:23.073 INFO MemoryStore - Block broadcast_494_piece0 stored as bytes in memory (estimated size 1890.0 B, free 1919.3 MiB)
17:38:23.073 INFO BlockManagerInfo - Added broadcast_494_piece0 in memory on localhost:37953 (size: 1890.0 B, free: 1919.9 MiB)
17:38:23.074 INFO SparkContext - Created broadcast 494 from broadcast at BamSink.java:76
17:38:23.076 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam.parts dst=null perm=null proto=rpc
17:38:23.076 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:23.076 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:23.076 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:23.077 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=mkdirs src=/user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam.parts/_temporary/0 dst=null perm=runner:supergroup:rwxr-xr-x proto=rpc
17:38:23.083 INFO SparkContext - Starting job: runJob at SparkHadoopWriter.scala:83
17:38:23.084 INFO DAGScheduler - Registering RDD 1180 (mapToPair at SparkUtils.java:161) as input to shuffle 49
17:38:23.084 INFO DAGScheduler - Got job 185 (runJob at SparkHadoopWriter.scala:83) with 1 output partitions
17:38:23.084 INFO DAGScheduler - Final stage: ResultStage 246 (runJob at SparkHadoopWriter.scala:83)
17:38:23.084 INFO DAGScheduler - Parents of final stage: List(ShuffleMapStage 245)
17:38:23.084 INFO DAGScheduler - Missing parents: List(ShuffleMapStage 245)
17:38:23.084 INFO DAGScheduler - Submitting ShuffleMapStage 245 (MapPartitionsRDD[1180] at mapToPair at SparkUtils.java:161), which has no missing parents
17:38:23.099 INFO MemoryStore - Block broadcast_495 stored as values in memory (estimated size 434.3 KiB, free 1918.9 MiB)
17:38:23.101 INFO MemoryStore - Block broadcast_495_piece0 stored as bytes in memory (estimated size 157.6 KiB, free 1918.7 MiB)
17:38:23.101 INFO BlockManagerInfo - Added broadcast_495_piece0 in memory on localhost:37953 (size: 157.6 KiB, free: 1919.7 MiB)
17:38:23.101 INFO SparkContext - Created broadcast 495 from broadcast at DAGScheduler.scala:1580
17:38:23.101 INFO DAGScheduler - Submitting 1 missing tasks from ShuffleMapStage 245 (MapPartitionsRDD[1180] at mapToPair at SparkUtils.java:161) (first 15 tasks are for partitions Vector(0))
17:38:23.101 INFO TaskSchedulerImpl - Adding task set 245.0 with 1 tasks resource profile 0
17:38:23.102 INFO TaskSetManager - Starting task 0.0 in stage 245.0 (TID 301) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7882 bytes)
17:38:23.102 INFO Executor - Running task 0.0 in stage 245.0 (TID 301)
17:38:23.135 INFO NewHadoopRDD - Input split: file:/home/runner/work/gatk/gatk/src/test/resources/org/broadinstitute/hellbender/tools/BQSR/CEUTrio.HiSeq.WGS.b37.ch20.1m-1m1k.NA12878.bam:0+211123
17:38:23.135 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:23.136 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:23.141 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:23.149 INFO Executor - Finished task 0.0 in stage 245.0 (TID 301). 1148 bytes result sent to driver
17:38:23.150 INFO TaskSetManager - Finished task 0.0 in stage 245.0 (TID 301) in 48 ms on localhost (executor driver) (1/1)
17:38:23.150 INFO TaskSchedulerImpl - Removed TaskSet 245.0, whose tasks have all completed, from pool
17:38:23.150 INFO DAGScheduler - ShuffleMapStage 245 (mapToPair at SparkUtils.java:161) finished in 0.066 s
17:38:23.150 INFO DAGScheduler - looking for newly runnable stages
17:38:23.150 INFO DAGScheduler - running: HashSet()
17:38:23.150 INFO DAGScheduler - waiting: HashSet(ResultStage 246)
17:38:23.150 INFO DAGScheduler - failed: HashSet()
17:38:23.150 INFO DAGScheduler - Submitting ResultStage 246 (MapPartitionsRDD[1185] at mapToPair at BamSink.java:91), which has no missing parents
17:38:23.156 INFO MemoryStore - Block broadcast_496 stored as values in memory (estimated size 155.4 KiB, free 1918.5 MiB)
17:38:23.157 INFO MemoryStore - Block broadcast_496_piece0 stored as bytes in memory (estimated size 58.5 KiB, free 1918.5 MiB)
17:38:23.157 INFO BlockManagerInfo - Added broadcast_496_piece0 in memory on localhost:37953 (size: 58.5 KiB, free: 1919.7 MiB)
17:38:23.157 INFO SparkContext - Created broadcast 496 from broadcast at DAGScheduler.scala:1580
17:38:23.157 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 246 (MapPartitionsRDD[1185] at mapToPair at BamSink.java:91) (first 15 tasks are for partitions Vector(0))
17:38:23.157 INFO TaskSchedulerImpl - Adding task set 246.0 with 1 tasks resource profile 0
17:38:23.158 INFO TaskSetManager - Starting task 0.0 in stage 246.0 (TID 302) (localhost, executor driver, partition 0, NODE_LOCAL, 7513 bytes)
17:38:23.158 INFO Executor - Running task 0.0 in stage 246.0 (TID 302)
17:38:23.162 INFO ShuffleBlockFetcherIterator - Getting 1 (312.6 KiB) non-empty blocks including 1 (312.6 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
17:38:23.163 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
17:38:23.173 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:23.173 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:23.173 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:23.174 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:23.174 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:23.174 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:23.175 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam.parts/_temporary/0/_temporary/attempt_202510271738232338364259884928896_1185_r_000000_0/part-r-00000 dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:23.175 WARN IntelDeflaterFactory - Intel Deflater not supported, using Java.util.zip.Deflater
17:38:23.176 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam.parts/_temporary/0/_temporary/attempt_202510271738232338364259884928896_1185_r_000000_0/.part-r-00000.sbi dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:23.177 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam.parts/_temporary/0/_temporary/attempt_202510271738232338364259884928896_1185_r_000000_0/.part-r-00000.bai dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:23.180 INFO StateChange - BLOCK* allocate blk_1073741905_1081, replicas=127.0.0.1:37927 for /user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam.parts/_temporary/0/_temporary/attempt_202510271738232338364259884928896_1185_r_000000_0/part-r-00000
17:38:23.181 INFO DataNode - Receiving BP-617366194-10.1.0.252-1761586664431:blk_1073741905_1081 src: /127.0.0.1:49720 dest: /127.0.0.1:37927
17:38:23.185 INFO clienttrace - src: /127.0.0.1:49720, dest: /127.0.0.1:37927, bytes: 235870, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1369393814_1, offset: 0, srvID: 379dd5e5-59c3-42cd-b9d3-67ca5e44560b, blockid: BP-617366194-10.1.0.252-1761586664431:blk_1073741905_1081, duration(ns): 3474869
17:38:23.185 INFO DataNode - PacketResponder: BP-617366194-10.1.0.252-1761586664431:blk_1073741905_1081, type=LAST_IN_PIPELINE terminating
17:38:23.186 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam.parts/_temporary/0/_temporary/attempt_202510271738232338364259884928896_1185_r_000000_0/part-r-00000 is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:38:23.186 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam.parts/_temporary/0/_temporary/attempt_202510271738232338364259884928896_1185_r_000000_0/part-r-00000 dst=null perm=null proto=rpc
17:38:23.187 INFO StateChange - BLOCK* allocate blk_1073741906_1082, replicas=127.0.0.1:37927 for /user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam.parts/_temporary/0/_temporary/attempt_202510271738232338364259884928896_1185_r_000000_0/.part-r-00000.sbi
17:38:23.188 INFO DataNode - Receiving BP-617366194-10.1.0.252-1761586664431:blk_1073741906_1082 src: /127.0.0.1:49734 dest: /127.0.0.1:37927
17:38:23.189 INFO clienttrace - src: /127.0.0.1:49734, dest: /127.0.0.1:37927, bytes: 204, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1369393814_1, offset: 0, srvID: 379dd5e5-59c3-42cd-b9d3-67ca5e44560b, blockid: BP-617366194-10.1.0.252-1761586664431:blk_1073741906_1082, duration(ns): 441869
17:38:23.189 INFO DataNode - PacketResponder: BP-617366194-10.1.0.252-1761586664431:blk_1073741906_1082, type=LAST_IN_PIPELINE terminating
17:38:23.189 INFO FSNamesystem - BLOCK* blk_1073741906_1082 is COMMITTED but not COMPLETE(numNodes= 0 < minimum = 1) in file /user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam.parts/_temporary/0/_temporary/attempt_202510271738232338364259884928896_1185_r_000000_0/.part-r-00000.sbi
17:38:23.590 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam.parts/_temporary/0/_temporary/attempt_202510271738232338364259884928896_1185_r_000000_0/.part-r-00000.sbi is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:38:23.591 INFO StateChange - BLOCK* allocate blk_1073741907_1083, replicas=127.0.0.1:37927 for /user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam.parts/_temporary/0/_temporary/attempt_202510271738232338364259884928896_1185_r_000000_0/.part-r-00000.bai
17:38:23.592 INFO DataNode - Receiving BP-617366194-10.1.0.252-1761586664431:blk_1073741907_1083 src: /127.0.0.1:49746 dest: /127.0.0.1:37927
17:38:23.594 INFO clienttrace - src: /127.0.0.1:49746, dest: /127.0.0.1:37927, bytes: 592, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1369393814_1, offset: 0, srvID: 379dd5e5-59c3-42cd-b9d3-67ca5e44560b, blockid: BP-617366194-10.1.0.252-1761586664431:blk_1073741907_1083, duration(ns): 499174
17:38:23.594 INFO DataNode - PacketResponder: BP-617366194-10.1.0.252-1761586664431:blk_1073741907_1083, type=LAST_IN_PIPELINE terminating
17:38:23.594 INFO FSNamesystem - BLOCK* blk_1073741907_1083 is COMMITTED but not COMPLETE(numNodes= 0 < minimum = 1) in file /user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam.parts/_temporary/0/_temporary/attempt_202510271738232338364259884928896_1185_r_000000_0/.part-r-00000.bai
17:38:23.995 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam.parts/_temporary/0/_temporary/attempt_202510271738232338364259884928896_1185_r_000000_0/.part-r-00000.bai is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:38:23.996 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam.parts/_temporary/0/_temporary/attempt_202510271738232338364259884928896_1185_r_000000_0 dst=null perm=null proto=rpc
17:38:23.997 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam.parts/_temporary/0/_temporary/attempt_202510271738232338364259884928896_1185_r_000000_0 dst=null perm=null proto=rpc
17:38:23.997 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam.parts/_temporary/0/task_202510271738232338364259884928896_1185_r_000000 dst=null perm=null proto=rpc
17:38:23.998 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=rename src=/user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam.parts/_temporary/0/_temporary/attempt_202510271738232338364259884928896_1185_r_000000_0 dst=/user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam.parts/_temporary/0/task_202510271738232338364259884928896_1185_r_000000 perm=runner:supergroup:rwxr-xr-x proto=rpc
17:38:23.998 INFO FileOutputCommitter - Saved output of task 'attempt_202510271738232338364259884928896_1185_r_000000_0' to hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam.parts/_temporary/0/task_202510271738232338364259884928896_1185_r_000000
17:38:23.998 INFO SparkHadoopMapRedUtil - attempt_202510271738232338364259884928896_1185_r_000000_0: Committed. Elapsed time: 1 ms.
17:38:23.999 INFO Executor - Finished task 0.0 in stage 246.0 (TID 302). 1858 bytes result sent to driver
17:38:23.999 INFO TaskSetManager - Finished task 0.0 in stage 246.0 (TID 302) in 841 ms on localhost (executor driver) (1/1)
17:38:23.999 INFO TaskSchedulerImpl - Removed TaskSet 246.0, whose tasks have all completed, from pool
17:38:23.999 INFO DAGScheduler - ResultStage 246 (runJob at SparkHadoopWriter.scala:83) finished in 0.849 s
17:38:23.999 INFO DAGScheduler - Job 185 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:24.000 INFO TaskSchedulerImpl - Killing all running tasks in stage 246: Stage finished
17:38:24.000 INFO DAGScheduler - Job 185 finished: runJob at SparkHadoopWriter.scala:83, took 0.916383 s
17:38:24.000 INFO SparkHadoopWriter - Start to commit write Job job_202510271738232338364259884928896_1185.
17:38:24.000 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=listStatus src=/user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam.parts/_temporary/0 dst=null perm=null proto=rpc
17:38:24.001 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam.parts dst=null perm=null proto=rpc
17:38:24.001 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=listStatus src=/user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam.parts/_temporary/0/task_202510271738232338364259884928896_1185_r_000000 dst=null perm=null proto=rpc
17:38:24.002 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam.parts/.part-r-00000.bai dst=null perm=null proto=rpc
17:38:24.002 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=rename src=/user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam.parts/_temporary/0/task_202510271738232338364259884928896_1185_r_000000/.part-r-00000.bai dst=/user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam.parts/.part-r-00000.bai perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:24.003 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam.parts/.part-r-00000.sbi dst=null perm=null proto=rpc
17:38:24.003 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=rename src=/user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam.parts/_temporary/0/task_202510271738232338364259884928896_1185_r_000000/.part-r-00000.sbi dst=/user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam.parts/.part-r-00000.sbi perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:24.004 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam.parts/part-r-00000 dst=null perm=null proto=rpc
17:38:24.004 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=rename src=/user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam.parts/_temporary/0/task_202510271738232338364259884928896_1185_r_000000/part-r-00000 dst=/user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam.parts/part-r-00000 perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:24.005 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=delete src=/user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam.parts/_temporary dst=null perm=null proto=rpc
17:38:24.005 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam.parts/_SUCCESS dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:24.006 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam.parts/_SUCCESS is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:38:24.007 INFO audit - allowed=false ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=delete src=/user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam.parts/.spark-staging-1185 dst=null perm=null proto=rpc
17:38:24.007 INFO SparkHadoopWriter - Write Job job_202510271738232338364259884928896_1185 committed. Elapsed time: 7 ms.
17:38:24.008 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam.parts/header dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:24.008 WARN IntelDeflaterFactory - Intel Deflater not supported, using Java.util.zip.Deflater
17:38:24.009 INFO StateChange - BLOCK* allocate blk_1073741908_1084, replicas=127.0.0.1:37927 for /user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam.parts/header
17:38:24.010 INFO DataNode - Receiving BP-617366194-10.1.0.252-1761586664431:blk_1073741908_1084 src: /127.0.0.1:49754 dest: /127.0.0.1:37927
17:38:24.011 INFO clienttrace - src: /127.0.0.1:49754, dest: /127.0.0.1:37927, bytes: 1180, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1369393814_1, offset: 0, srvID: 379dd5e5-59c3-42cd-b9d3-67ca5e44560b, blockid: BP-617366194-10.1.0.252-1761586664431:blk_1073741908_1084, duration(ns): 528992
17:38:24.011 INFO DataNode - PacketResponder: BP-617366194-10.1.0.252-1761586664431:blk_1073741908_1084, type=LAST_IN_PIPELINE terminating
17:38:24.012 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam.parts/header is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:38:24.012 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam.parts/terminator dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:24.013 INFO StateChange - BLOCK* allocate blk_1073741909_1085, replicas=127.0.0.1:37927 for /user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam.parts/terminator
17:38:24.014 INFO DataNode - Receiving BP-617366194-10.1.0.252-1761586664431:blk_1073741909_1085 src: /127.0.0.1:49758 dest: /127.0.0.1:37927
17:38:24.015 INFO clienttrace - src: /127.0.0.1:49758, dest: /127.0.0.1:37927, bytes: 28, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1369393814_1, offset: 0, srvID: 379dd5e5-59c3-42cd-b9d3-67ca5e44560b, blockid: BP-617366194-10.1.0.252-1761586664431:blk_1073741909_1085, duration(ns): 459821
17:38:24.015 INFO DataNode - PacketResponder: BP-617366194-10.1.0.252-1761586664431:blk_1073741909_1085, type=LAST_IN_PIPELINE terminating
17:38:24.015 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam.parts/terminator is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:38:24.016 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=listStatus src=/user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam.parts dst=null perm=null proto=rpc
17:38:24.017 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam.parts/output dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:24.018 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam.parts/output is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:38:24.018 INFO HadoopFileSystemWrapper - Concatenating 3 parts to hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam
17:38:24.019 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=concat src=[/user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam.parts/header, /user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam.parts/part-r-00000, /user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam.parts/terminator] dst=/user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam.parts/output perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:24.019 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam dst=null perm=null proto=rpc
17:38:24.020 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=delete src=/user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam dst=null perm=null proto=rpc
17:38:24.020 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=rename src=/user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam.parts/output dst=/user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:24.020 INFO HadoopFileSystemWrapper - Concatenating to hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam done
17:38:24.020 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam dst=null perm=null proto=rpc
17:38:24.021 INFO IndexFileMerger - Merging .sbi files in temp directory hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam.parts/ to hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam.sbi
17:38:24.021 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=listStatus src=/user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam.parts dst=null perm=null proto=rpc
17:38:24.022 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam.sbi dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:24.023 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam.parts/.part-r-00000.sbi dst=null perm=null proto=rpc
17:38:24.023 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam.parts/.part-r-00000.sbi dst=null perm=null proto=rpc
17:38:24.024 WARN DFSUtil - Unexpected value for data transfer bytes=208 duration=0
17:38:24.024 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=delete src=/user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam.parts/.part-r-00000.sbi dst=null perm=null proto=rpc
17:38:24.025 INFO StateChange - BLOCK* allocate blk_1073741910_1086, replicas=127.0.0.1:37927 for /user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam.sbi
17:38:24.026 INFO DataNode - Receiving BP-617366194-10.1.0.252-1761586664431:blk_1073741910_1086 src: /127.0.0.1:49760 dest: /127.0.0.1:37927
17:38:24.027 INFO clienttrace - src: /127.0.0.1:49760, dest: /127.0.0.1:37927, bytes: 204, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1369393814_1, offset: 0, srvID: 379dd5e5-59c3-42cd-b9d3-67ca5e44560b, blockid: BP-617366194-10.1.0.252-1761586664431:blk_1073741910_1086, duration(ns): 432533
17:38:24.027 INFO DataNode - PacketResponder: BP-617366194-10.1.0.252-1761586664431:blk_1073741910_1086, type=LAST_IN_PIPELINE terminating
17:38:24.028 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam.sbi is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:38:24.028 INFO IndexFileMerger - Done merging .sbi files
17:38:24.028 INFO IndexFileMerger - Merging .bai files in temp directory hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam.parts/ to hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam.bai
17:38:24.028 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=listStatus src=/user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam.parts dst=null perm=null proto=rpc
17:38:24.029 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam.bai dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:24.030 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam.parts/.part-r-00000.bai dst=null perm=null proto=rpc
17:38:24.030 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam.parts/.part-r-00000.bai dst=null perm=null proto=rpc
17:38:24.031 WARN DFSUtil - Unexpected value for data transfer bytes=600 duration=0
17:38:24.031 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=delete src=/user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam.parts/.part-r-00000.bai dst=null perm=null proto=rpc
17:38:24.032 INFO StateChange - BLOCK* allocate blk_1073741911_1087, replicas=127.0.0.1:37927 for /user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam.bai
17:38:24.033 INFO DataNode - Receiving BP-617366194-10.1.0.252-1761586664431:blk_1073741911_1087 src: /127.0.0.1:49770 dest: /127.0.0.1:37927
17:38:24.034 INFO clienttrace - src: /127.0.0.1:49770, dest: /127.0.0.1:37927, bytes: 592, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1369393814_1, offset: 0, srvID: 379dd5e5-59c3-42cd-b9d3-67ca5e44560b, blockid: BP-617366194-10.1.0.252-1761586664431:blk_1073741911_1087, duration(ns): 436637
17:38:24.034 INFO DataNode - PacketResponder: BP-617366194-10.1.0.252-1761586664431:blk_1073741911_1087, type=LAST_IN_PIPELINE terminating
17:38:24.034 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam.bai is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:38:24.035 INFO IndexFileMerger - Done merging .bai files
17:38:24.035 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=delete src=/user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam.parts dst=null perm=null proto=rpc
17:38:24.044 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam.bai dst=null perm=null proto=rpc
17:38:24.051 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam.sbi dst=null perm=null proto=rpc
17:38:24.052 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam.sbi dst=null perm=null proto=rpc
17:38:24.052 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam.sbi dst=null perm=null proto=rpc
17:38:24.053 WARN DFSUtil - Unexpected value for data transfer bytes=208 duration=0
17:38:24.054 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam dst=null perm=null proto=rpc
17:38:24.054 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam dst=null perm=null proto=rpc
17:38:24.055 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam dst=null perm=null proto=rpc
17:38:24.055 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam dst=null perm=null proto=rpc
17:38:24.056 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam.bai dst=null perm=null proto=rpc
17:38:24.056 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam.bai dst=null perm=null proto=rpc
17:38:24.057 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam.bai dst=null perm=null proto=rpc
17:38:24.057 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:24.058 WARN DFSUtil - Unexpected value for data transfer bytes=1192 duration=0
17:38:24.059 WARN DFSUtil - Unexpected value for data transfer bytes=600 duration=0
17:38:24.060 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam.sbi dst=null perm=null proto=rpc
17:38:24.060 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam.sbi dst=null perm=null proto=rpc
17:38:24.061 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam.sbi dst=null perm=null proto=rpc
17:38:24.061 WARN DFSUtil - Unexpected value for data transfer bytes=208 duration=0
17:38:24.062 INFO MemoryStore - Block broadcast_497 stored as values in memory (estimated size 312.0 B, free 1918.5 MiB)
17:38:24.062 INFO MemoryStore - Block broadcast_497_piece0 stored as bytes in memory (estimated size 231.0 B, free 1918.5 MiB)
17:38:24.062 INFO BlockManagerInfo - Added broadcast_497_piece0 in memory on localhost:37953 (size: 231.0 B, free: 1919.7 MiB)
17:38:24.062 INFO SparkContext - Created broadcast 497 from broadcast at BamSource.java:104
17:38:24.063 INFO MemoryStore - Block broadcast_498 stored as values in memory (estimated size 297.9 KiB, free 1918.2 MiB)
17:38:24.068 INFO BlockManagerInfo - Removed broadcast_496_piece0 on localhost:37953 in memory (size: 58.5 KiB, free: 1919.7 MiB)
17:38:24.069 INFO BlockManagerInfo - Removed broadcast_493_piece0 on localhost:37953 in memory (size: 1890.0 B, free: 1919.7 MiB)
17:38:24.069 INFO BlockManagerInfo - Removed broadcast_492_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.8 MiB)
17:38:24.070 INFO BlockManagerInfo - Removed broadcast_494_piece0 on localhost:37953 in memory (size: 1890.0 B, free: 1919.8 MiB)
17:38:24.071 INFO BlockManagerInfo - Removed broadcast_495_piece0 on localhost:37953 in memory (size: 157.6 KiB, free: 1920.0 MiB)
17:38:24.072 INFO MemoryStore - Block broadcast_498_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1919.3 MiB)
17:38:24.072 INFO BlockManagerInfo - Added broadcast_498_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.9 MiB)
17:38:24.072 INFO SparkContext - Created broadcast 498 from newAPIHadoopFile at PathSplitSource.java:96
17:38:24.081 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam dst=null perm=null proto=rpc
17:38:24.081 INFO FileInputFormat - Total input files to process : 1
17:38:24.082 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam dst=null perm=null proto=rpc
17:38:24.095 INFO SparkContext - Starting job: collect at ReadsSparkSinkUnitTest.java:182
17:38:24.095 INFO DAGScheduler - Got job 186 (collect at ReadsSparkSinkUnitTest.java:182) with 1 output partitions
17:38:24.095 INFO DAGScheduler - Final stage: ResultStage 247 (collect at ReadsSparkSinkUnitTest.java:182)
17:38:24.095 INFO DAGScheduler - Parents of final stage: List()
17:38:24.095 INFO DAGScheduler - Missing parents: List()
17:38:24.096 INFO DAGScheduler - Submitting ResultStage 247 (MapPartitionsRDD[1191] at filter at ReadsSparkSource.java:96), which has no missing parents
17:38:24.101 INFO MemoryStore - Block broadcast_499 stored as values in memory (estimated size 148.2 KiB, free 1919.2 MiB)
17:38:24.102 INFO MemoryStore - Block broadcast_499_piece0 stored as bytes in memory (estimated size 54.6 KiB, free 1919.1 MiB)
17:38:24.102 INFO BlockManagerInfo - Added broadcast_499_piece0 in memory on localhost:37953 (size: 54.6 KiB, free: 1919.8 MiB)
17:38:24.102 INFO SparkContext - Created broadcast 499 from broadcast at DAGScheduler.scala:1580
17:38:24.102 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 247 (MapPartitionsRDD[1191] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:38:24.102 INFO TaskSchedulerImpl - Adding task set 247.0 with 1 tasks resource profile 0
17:38:24.103 INFO TaskSetManager - Starting task 0.0 in stage 247.0 (TID 303) (localhost, executor driver, partition 0, ANY, 7852 bytes)
17:38:24.103 INFO Executor - Running task 0.0 in stage 247.0 (TID 303)
17:38:24.115 INFO NewHadoopRDD - Input split: hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam:0+237078
17:38:24.116 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam dst=null perm=null proto=rpc
17:38:24.117 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam dst=null perm=null proto=rpc
17:38:24.117 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam.bai dst=null perm=null proto=rpc
17:38:24.118 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam.bai dst=null perm=null proto=rpc
17:38:24.118 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam.bai dst=null perm=null proto=rpc
17:38:24.119 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:24.120 WARN DFSUtil - Unexpected value for data transfer bytes=1192 duration=0
17:38:24.121 WARN DFSUtil - Unexpected value for data transfer bytes=600 duration=0
17:38:24.122 WARN DFSUtil - Unexpected value for data transfer bytes=600 duration=0
17:38:24.123 WARN DFSUtil - Unexpected value for data transfer bytes=237714 duration=0
17:38:24.123 WARN DFSUtil - Unexpected value for data transfer bytes=32 duration=0
17:38:24.125 INFO Executor - Finished task 0.0 in stage 247.0 (TID 303). 749470 bytes result sent to driver
17:38:24.127 INFO TaskSetManager - Finished task 0.0 in stage 247.0 (TID 303) in 24 ms on localhost (executor driver) (1/1)
17:38:24.127 INFO TaskSchedulerImpl - Removed TaskSet 247.0, whose tasks have all completed, from pool
17:38:24.127 INFO DAGScheduler - ResultStage 247 (collect at ReadsSparkSinkUnitTest.java:182) finished in 0.031 s
17:38:24.127 INFO DAGScheduler - Job 186 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:24.127 INFO TaskSchedulerImpl - Killing all running tasks in stage 247: Stage finished
17:38:24.127 INFO DAGScheduler - Job 186 finished: collect at ReadsSparkSinkUnitTest.java:182, took 0.032051 s
17:38:24.138 INFO SparkContext - Starting job: count at ReadsSparkSinkUnitTest.java:185
17:38:24.138 INFO DAGScheduler - Got job 187 (count at ReadsSparkSinkUnitTest.java:185) with 1 output partitions
17:38:24.138 INFO DAGScheduler - Final stage: ResultStage 248 (count at ReadsSparkSinkUnitTest.java:185)
17:38:24.138 INFO DAGScheduler - Parents of final stage: List()
17:38:24.139 INFO DAGScheduler - Missing parents: List()
17:38:24.139 INFO DAGScheduler - Submitting ResultStage 248 (MapPartitionsRDD[1173] at filter at ReadsSparkSource.java:96), which has no missing parents
17:38:24.154 INFO MemoryStore - Block broadcast_500 stored as values in memory (estimated size 426.1 KiB, free 1918.7 MiB)
17:38:24.155 INFO MemoryStore - Block broadcast_500_piece0 stored as bytes in memory (estimated size 153.6 KiB, free 1918.6 MiB)
17:38:24.155 INFO BlockManagerInfo - Added broadcast_500_piece0 in memory on localhost:37953 (size: 153.6 KiB, free: 1919.7 MiB)
17:38:24.155 INFO SparkContext - Created broadcast 500 from broadcast at DAGScheduler.scala:1580
17:38:24.155 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 248 (MapPartitionsRDD[1173] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:38:24.155 INFO TaskSchedulerImpl - Adding task set 248.0 with 1 tasks resource profile 0
17:38:24.156 INFO TaskSetManager - Starting task 0.0 in stage 248.0 (TID 304) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7893 bytes)
17:38:24.156 INFO Executor - Running task 0.0 in stage 248.0 (TID 304)
17:38:24.187 INFO NewHadoopRDD - Input split: file:/home/runner/work/gatk/gatk/src/test/resources/org/broadinstitute/hellbender/tools/BQSR/CEUTrio.HiSeq.WGS.b37.ch20.1m-1m1k.NA12878.bam:0+211123
17:38:24.188 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:24.188 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:24.194 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:24.197 INFO Executor - Finished task 0.0 in stage 248.0 (TID 304). 989 bytes result sent to driver
17:38:24.197 INFO TaskSetManager - Finished task 0.0 in stage 248.0 (TID 304) in 41 ms on localhost (executor driver) (1/1)
17:38:24.197 INFO TaskSchedulerImpl - Removed TaskSet 248.0, whose tasks have all completed, from pool
17:38:24.197 INFO DAGScheduler - ResultStage 248 (count at ReadsSparkSinkUnitTest.java:185) finished in 0.058 s
17:38:24.198 INFO DAGScheduler - Job 187 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:24.198 INFO TaskSchedulerImpl - Killing all running tasks in stage 248: Stage finished
17:38:24.198 INFO DAGScheduler - Job 187 finished: count at ReadsSparkSinkUnitTest.java:185, took 0.059428 s
17:38:24.201 INFO SparkContext - Starting job: count at ReadsSparkSinkUnitTest.java:185
17:38:24.201 INFO DAGScheduler - Got job 188 (count at ReadsSparkSinkUnitTest.java:185) with 1 output partitions
17:38:24.201 INFO DAGScheduler - Final stage: ResultStage 249 (count at ReadsSparkSinkUnitTest.java:185)
17:38:24.201 INFO DAGScheduler - Parents of final stage: List()
17:38:24.201 INFO DAGScheduler - Missing parents: List()
17:38:24.201 INFO DAGScheduler - Submitting ResultStage 249 (MapPartitionsRDD[1191] at filter at ReadsSparkSource.java:96), which has no missing parents
17:38:24.207 INFO MemoryStore - Block broadcast_501 stored as values in memory (estimated size 148.1 KiB, free 1918.4 MiB)
17:38:24.208 INFO MemoryStore - Block broadcast_501_piece0 stored as bytes in memory (estimated size 54.6 KiB, free 1918.4 MiB)
17:38:24.208 INFO BlockManagerInfo - Added broadcast_501_piece0 in memory on localhost:37953 (size: 54.6 KiB, free: 1919.6 MiB)
17:38:24.208 INFO SparkContext - Created broadcast 501 from broadcast at DAGScheduler.scala:1580
17:38:24.208 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 249 (MapPartitionsRDD[1191] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:38:24.208 INFO TaskSchedulerImpl - Adding task set 249.0 with 1 tasks resource profile 0
17:38:24.209 INFO TaskSetManager - Starting task 0.0 in stage 249.0 (TID 305) (localhost, executor driver, partition 0, ANY, 7852 bytes)
17:38:24.209 INFO Executor - Running task 0.0 in stage 249.0 (TID 305)
17:38:24.220 INFO NewHadoopRDD - Input split: hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam:0+237078
17:38:24.221 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam dst=null perm=null proto=rpc
17:38:24.222 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam dst=null perm=null proto=rpc
17:38:24.222 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam.bai dst=null perm=null proto=rpc
17:38:24.223 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam.bai dst=null perm=null proto=rpc
17:38:24.223 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest3_459ce17d-d44a-4238-a350-688bc6eb012d.bam.bai dst=null perm=null proto=rpc
17:38:24.224 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:24.225 WARN DFSUtil - Unexpected value for data transfer bytes=1192 duration=0
17:38:24.226 WARN DFSUtil - Unexpected value for data transfer bytes=600 duration=0
17:38:24.226 WARN DFSUtil - Unexpected value for data transfer bytes=600 duration=0
17:38:24.227 WARN DFSUtil - Unexpected value for data transfer bytes=237714 duration=0
17:38:24.228 WARN DFSUtil - Unexpected value for data transfer bytes=32 duration=0
17:38:24.229 INFO Executor - Finished task 0.0 in stage 249.0 (TID 305). 989 bytes result sent to driver
17:38:24.230 INFO TaskSetManager - Finished task 0.0 in stage 249.0 (TID 305) in 22 ms on localhost (executor driver) (1/1)
17:38:24.230 INFO TaskSchedulerImpl - Removed TaskSet 249.0, whose tasks have all completed, from pool
17:38:24.230 INFO DAGScheduler - ResultStage 249 (count at ReadsSparkSinkUnitTest.java:185) finished in 0.029 s
17:38:24.230 INFO DAGScheduler - Job 188 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:24.230 INFO TaskSchedulerImpl - Killing all running tasks in stage 249: Stage finished
17:38:24.230 INFO DAGScheduler - Job 188 finished: count at ReadsSparkSinkUnitTest.java:185, took 0.029291 s
17:38:24.238 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest5_3a6038d6-4929-498a-89bd-0e9fb758a8e6.cram dst=null perm=null proto=rpc
17:38:24.238 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest5_3a6038d6-4929-498a-89bd-0e9fb758a8e6.cram dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:24.239 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest5_3a6038d6-4929-498a-89bd-0e9fb758a8e6.cram is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:38:24.239 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest5_3a6038d6-4929-498a-89bd-0e9fb758a8e6.cram dst=null perm=null proto=rpc
17:38:24.242 INFO MemoryStore - Block broadcast_502 stored as values in memory (estimated size 576.0 B, free 1918.4 MiB)
17:38:24.242 INFO MemoryStore - Block broadcast_502_piece0 stored as bytes in memory (estimated size 228.0 B, free 1918.4 MiB)
17:38:24.242 INFO BlockManagerInfo - Added broadcast_502_piece0 in memory on localhost:37953 (size: 228.0 B, free: 1919.6 MiB)
17:38:24.242 INFO SparkContext - Created broadcast 502 from broadcast at CramSource.java:114
17:38:24.243 INFO MemoryStore - Block broadcast_503 stored as values in memory (estimated size 297.9 KiB, free 1918.1 MiB)
17:38:24.249 INFO MemoryStore - Block broadcast_503_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1918.0 MiB)
17:38:24.249 INFO BlockManagerInfo - Added broadcast_503_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.6 MiB)
17:38:24.249 INFO SparkContext - Created broadcast 503 from newAPIHadoopFile at PathSplitSource.java:96
17:38:24.264 INFO MemoryStore - Block broadcast_504 stored as values in memory (estimated size 576.0 B, free 1918.0 MiB)
17:38:24.267 INFO MemoryStore - Block broadcast_504_piece0 stored as bytes in memory (estimated size 228.0 B, free 1918.0 MiB)
17:38:24.267 INFO BlockManagerInfo - Added broadcast_504_piece0 in memory on localhost:37953 (size: 228.0 B, free: 1919.6 MiB)
17:38:24.267 INFO BlockManagerInfo - Removed broadcast_499_piece0 on localhost:37953 in memory (size: 54.6 KiB, free: 1919.6 MiB)
17:38:24.268 INFO SparkContext - Created broadcast 504 from broadcast at CramSource.java:114
17:38:24.268 INFO BlockManagerInfo - Removed broadcast_497_piece0 on localhost:37953 in memory (size: 231.0 B, free: 1919.6 MiB)
17:38:24.268 INFO BlockManagerInfo - Removed broadcast_501_piece0 on localhost:37953 in memory (size: 54.6 KiB, free: 1919.7 MiB)
17:38:24.268 INFO MemoryStore - Block broadcast_505 stored as values in memory (estimated size 297.9 KiB, free 1918.1 MiB)
17:38:24.269 INFO BlockManagerInfo - Removed broadcast_491_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.8 MiB)
17:38:24.269 INFO BlockManagerInfo - Removed broadcast_500_piece0 on localhost:37953 in memory (size: 153.6 KiB, free: 1919.9 MiB)
17:38:24.269 INFO BlockManagerInfo - Removed broadcast_498_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1920.0 MiB)
17:38:24.274 INFO MemoryStore - Block broadcast_505_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1919.3 MiB)
17:38:24.274 INFO BlockManagerInfo - Added broadcast_505_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.9 MiB)
17:38:24.274 INFO SparkContext - Created broadcast 505 from newAPIHadoopFile at PathSplitSource.java:96
17:38:24.288 INFO FileInputFormat - Total input files to process : 1
17:38:24.289 INFO MemoryStore - Block broadcast_506 stored as values in memory (estimated size 6.0 KiB, free 1919.3 MiB)
17:38:24.289 INFO MemoryStore - Block broadcast_506_piece0 stored as bytes in memory (estimated size 1473.0 B, free 1919.3 MiB)
17:38:24.289 INFO BlockManagerInfo - Added broadcast_506_piece0 in memory on localhost:37953 (size: 1473.0 B, free: 1919.9 MiB)
17:38:24.290 INFO SparkContext - Created broadcast 506 from broadcast at ReadsSparkSink.java:133
17:38:24.290 INFO MemoryStore - Block broadcast_507 stored as values in memory (estimated size 6.2 KiB, free 1919.3 MiB)
17:38:24.291 INFO MemoryStore - Block broadcast_507_piece0 stored as bytes in memory (estimated size 1473.0 B, free 1919.3 MiB)
17:38:24.291 INFO BlockManagerInfo - Added broadcast_507_piece0 in memory on localhost:37953 (size: 1473.0 B, free: 1919.9 MiB)
17:38:24.291 INFO SparkContext - Created broadcast 507 from broadcast at CramSink.java:76
17:38:24.293 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest5_3a6038d6-4929-498a-89bd-0e9fb758a8e6.cram.parts dst=null perm=null proto=rpc
17:38:24.293 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:24.293 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:24.293 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:24.294 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=mkdirs src=/user/runner/ReadsSparkSinkUnitTest5_3a6038d6-4929-498a-89bd-0e9fb758a8e6.cram.parts/_temporary/0 dst=null perm=runner:supergroup:rwxr-xr-x proto=rpc
17:38:24.300 INFO SparkContext - Starting job: runJob at SparkHadoopWriter.scala:83
17:38:24.300 INFO DAGScheduler - Registering RDD 1203 (mapToPair at SparkUtils.java:161) as input to shuffle 50
17:38:24.300 INFO DAGScheduler - Got job 189 (runJob at SparkHadoopWriter.scala:83) with 1 output partitions
17:38:24.300 INFO DAGScheduler - Final stage: ResultStage 251 (runJob at SparkHadoopWriter.scala:83)
17:38:24.300 INFO DAGScheduler - Parents of final stage: List(ShuffleMapStage 250)
17:38:24.300 INFO DAGScheduler - Missing parents: List(ShuffleMapStage 250)
17:38:24.301 INFO DAGScheduler - Submitting ShuffleMapStage 250 (MapPartitionsRDD[1203] at mapToPair at SparkUtils.java:161), which has no missing parents
17:38:24.311 INFO MemoryStore - Block broadcast_508 stored as values in memory (estimated size 292.8 KiB, free 1919.0 MiB)
17:38:24.312 INFO MemoryStore - Block broadcast_508_piece0 stored as bytes in memory (estimated size 107.3 KiB, free 1918.9 MiB)
17:38:24.312 INFO BlockManagerInfo - Added broadcast_508_piece0 in memory on localhost:37953 (size: 107.3 KiB, free: 1919.8 MiB)
17:38:24.312 INFO SparkContext - Created broadcast 508 from broadcast at DAGScheduler.scala:1580
17:38:24.312 INFO DAGScheduler - Submitting 1 missing tasks from ShuffleMapStage 250 (MapPartitionsRDD[1203] at mapToPair at SparkUtils.java:161) (first 15 tasks are for partitions Vector(0))
17:38:24.312 INFO TaskSchedulerImpl - Adding task set 250.0 with 1 tasks resource profile 0
17:38:24.313 INFO TaskSetManager - Starting task 0.0 in stage 250.0 (TID 306) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7869 bytes)
17:38:24.313 INFO Executor - Running task 0.0 in stage 250.0 (TID 306)
17:38:24.335 INFO NewHadoopRDD - Input split: file:/home/runner/work/gatk/gatk/src/test/resources/org/broadinstitute/hellbender/tools/BQSR/NA12878.chr17_69k_70k.dictFix.cram:0+50619
17:38:24.344 INFO Executor - Finished task 0.0 in stage 250.0 (TID 306). 1148 bytes result sent to driver
17:38:24.345 INFO TaskSetManager - Finished task 0.0 in stage 250.0 (TID 306) in 32 ms on localhost (executor driver) (1/1)
17:38:24.345 INFO TaskSchedulerImpl - Removed TaskSet 250.0, whose tasks have all completed, from pool
17:38:24.345 INFO DAGScheduler - ShuffleMapStage 250 (mapToPair at SparkUtils.java:161) finished in 0.044 s
17:38:24.345 INFO DAGScheduler - looking for newly runnable stages
17:38:24.345 INFO DAGScheduler - running: HashSet()
17:38:24.345 INFO DAGScheduler - waiting: HashSet(ResultStage 251)
17:38:24.345 INFO DAGScheduler - failed: HashSet()
17:38:24.345 INFO DAGScheduler - Submitting ResultStage 251 (MapPartitionsRDD[1208] at mapToPair at CramSink.java:89), which has no missing parents
17:38:24.351 INFO MemoryStore - Block broadcast_509 stored as values in memory (estimated size 153.3 KiB, free 1918.8 MiB)
17:38:24.352 INFO MemoryStore - Block broadcast_509_piece0 stored as bytes in memory (estimated size 58.1 KiB, free 1918.7 MiB)
17:38:24.352 INFO BlockManagerInfo - Added broadcast_509_piece0 in memory on localhost:37953 (size: 58.1 KiB, free: 1919.7 MiB)
17:38:24.352 INFO SparkContext - Created broadcast 509 from broadcast at DAGScheduler.scala:1580
17:38:24.352 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 251 (MapPartitionsRDD[1208] at mapToPair at CramSink.java:89) (first 15 tasks are for partitions Vector(0))
17:38:24.352 INFO TaskSchedulerImpl - Adding task set 251.0 with 1 tasks resource profile 0
17:38:24.353 INFO TaskSetManager - Starting task 0.0 in stage 251.0 (TID 307) (localhost, executor driver, partition 0, NODE_LOCAL, 7513 bytes)
17:38:24.353 INFO Executor - Running task 0.0 in stage 251.0 (TID 307)
17:38:24.357 INFO ShuffleBlockFetcherIterator - Getting 1 (82.3 KiB) non-empty blocks including 1 (82.3 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
17:38:24.357 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
17:38:24.364 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:24.364 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:24.364 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:24.364 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:24.364 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:24.364 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:24.365 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest5_3a6038d6-4929-498a-89bd-0e9fb758a8e6.cram.parts/_temporary/0/_temporary/attempt_202510271738245812104245629072461_1208_r_000000_0/part-r-00000 dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:24.386 INFO StateChange - BLOCK* allocate blk_1073741912_1088, replicas=127.0.0.1:37927 for /user/runner/ReadsSparkSinkUnitTest5_3a6038d6-4929-498a-89bd-0e9fb758a8e6.cram.parts/_temporary/0/_temporary/attempt_202510271738245812104245629072461_1208_r_000000_0/part-r-00000
17:38:24.387 INFO DataNode - Receiving BP-617366194-10.1.0.252-1761586664431:blk_1073741912_1088 src: /127.0.0.1:49786 dest: /127.0.0.1:37927
17:38:24.389 INFO clienttrace - src: /127.0.0.1:49786, dest: /127.0.0.1:37927, bytes: 42659, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1369393814_1, offset: 0, srvID: 379dd5e5-59c3-42cd-b9d3-67ca5e44560b, blockid: BP-617366194-10.1.0.252-1761586664431:blk_1073741912_1088, duration(ns): 560416
17:38:24.389 INFO DataNode - PacketResponder: BP-617366194-10.1.0.252-1761586664431:blk_1073741912_1088, type=LAST_IN_PIPELINE terminating
17:38:24.389 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest5_3a6038d6-4929-498a-89bd-0e9fb758a8e6.cram.parts/_temporary/0/_temporary/attempt_202510271738245812104245629072461_1208_r_000000_0/part-r-00000 is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:38:24.390 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest5_3a6038d6-4929-498a-89bd-0e9fb758a8e6.cram.parts/_temporary/0/_temporary/attempt_202510271738245812104245629072461_1208_r_000000_0 dst=null perm=null proto=rpc
17:38:24.391 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest5_3a6038d6-4929-498a-89bd-0e9fb758a8e6.cram.parts/_temporary/0/_temporary/attempt_202510271738245812104245629072461_1208_r_000000_0 dst=null perm=null proto=rpc
17:38:24.391 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest5_3a6038d6-4929-498a-89bd-0e9fb758a8e6.cram.parts/_temporary/0/task_202510271738245812104245629072461_1208_r_000000 dst=null perm=null proto=rpc
17:38:24.392 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=rename src=/user/runner/ReadsSparkSinkUnitTest5_3a6038d6-4929-498a-89bd-0e9fb758a8e6.cram.parts/_temporary/0/_temporary/attempt_202510271738245812104245629072461_1208_r_000000_0 dst=/user/runner/ReadsSparkSinkUnitTest5_3a6038d6-4929-498a-89bd-0e9fb758a8e6.cram.parts/_temporary/0/task_202510271738245812104245629072461_1208_r_000000 perm=runner:supergroup:rwxr-xr-x proto=rpc
17:38:24.392 INFO FileOutputCommitter - Saved output of task 'attempt_202510271738245812104245629072461_1208_r_000000_0' to hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest5_3a6038d6-4929-498a-89bd-0e9fb758a8e6.cram.parts/_temporary/0/task_202510271738245812104245629072461_1208_r_000000
17:38:24.392 INFO SparkHadoopMapRedUtil - attempt_202510271738245812104245629072461_1208_r_000000_0: Committed. Elapsed time: 1 ms.
17:38:24.393 INFO Executor - Finished task 0.0 in stage 251.0 (TID 307). 1858 bytes result sent to driver
17:38:24.393 INFO TaskSetManager - Finished task 0.0 in stage 251.0 (TID 307) in 40 ms on localhost (executor driver) (1/1)
17:38:24.393 INFO TaskSchedulerImpl - Removed TaskSet 251.0, whose tasks have all completed, from pool
17:38:24.393 INFO DAGScheduler - ResultStage 251 (runJob at SparkHadoopWriter.scala:83) finished in 0.047 s
17:38:24.393 INFO DAGScheduler - Job 189 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:24.393 INFO TaskSchedulerImpl - Killing all running tasks in stage 251: Stage finished
17:38:24.394 INFO DAGScheduler - Job 189 finished: runJob at SparkHadoopWriter.scala:83, took 0.093788 s
17:38:24.394 INFO SparkHadoopWriter - Start to commit write Job job_202510271738245812104245629072461_1208.
17:38:24.394 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=listStatus src=/user/runner/ReadsSparkSinkUnitTest5_3a6038d6-4929-498a-89bd-0e9fb758a8e6.cram.parts/_temporary/0 dst=null perm=null proto=rpc
17:38:24.395 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest5_3a6038d6-4929-498a-89bd-0e9fb758a8e6.cram.parts dst=null perm=null proto=rpc
17:38:24.395 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=listStatus src=/user/runner/ReadsSparkSinkUnitTest5_3a6038d6-4929-498a-89bd-0e9fb758a8e6.cram.parts/_temporary/0/task_202510271738245812104245629072461_1208_r_000000 dst=null perm=null proto=rpc
17:38:24.396 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest5_3a6038d6-4929-498a-89bd-0e9fb758a8e6.cram.parts/part-r-00000 dst=null perm=null proto=rpc
17:38:24.396 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=rename src=/user/runner/ReadsSparkSinkUnitTest5_3a6038d6-4929-498a-89bd-0e9fb758a8e6.cram.parts/_temporary/0/task_202510271738245812104245629072461_1208_r_000000/part-r-00000 dst=/user/runner/ReadsSparkSinkUnitTest5_3a6038d6-4929-498a-89bd-0e9fb758a8e6.cram.parts/part-r-00000 perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:24.397 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=delete src=/user/runner/ReadsSparkSinkUnitTest5_3a6038d6-4929-498a-89bd-0e9fb758a8e6.cram.parts/_temporary dst=null perm=null proto=rpc
17:38:24.397 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest5_3a6038d6-4929-498a-89bd-0e9fb758a8e6.cram.parts/_SUCCESS dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:24.398 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest5_3a6038d6-4929-498a-89bd-0e9fb758a8e6.cram.parts/_SUCCESS is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:38:24.399 INFO audit - allowed=false ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=delete src=/user/runner/ReadsSparkSinkUnitTest5_3a6038d6-4929-498a-89bd-0e9fb758a8e6.cram.parts/.spark-staging-1208 dst=null perm=null proto=rpc
17:38:24.399 INFO SparkHadoopWriter - Write Job job_202510271738245812104245629072461_1208 committed. Elapsed time: 5 ms.
17:38:24.399 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest5_3a6038d6-4929-498a-89bd-0e9fb758a8e6.cram.parts/header dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:24.404 INFO BlockManagerInfo - Removed broadcast_504_piece0 on localhost:37953 in memory (size: 228.0 B, free: 1919.7 MiB)
17:38:24.404 INFO StateChange - BLOCK* allocate blk_1073741913_1089, replicas=127.0.0.1:37927 for /user/runner/ReadsSparkSinkUnitTest5_3a6038d6-4929-498a-89bd-0e9fb758a8e6.cram.parts/header
17:38:24.405 INFO DataNode - Receiving BP-617366194-10.1.0.252-1761586664431:blk_1073741913_1089 src: /127.0.0.1:49798 dest: /127.0.0.1:37927
17:38:24.405 INFO BlockManagerInfo - Removed broadcast_508_piece0 on localhost:37953 in memory (size: 107.3 KiB, free: 1919.8 MiB)
17:38:24.405 INFO BlockManagerInfo - Removed broadcast_509_piece0 on localhost:37953 in memory (size: 58.1 KiB, free: 1919.9 MiB)
17:38:24.406 INFO clienttrace - src: /127.0.0.1:49798, dest: /127.0.0.1:37927, bytes: 1016, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1369393814_1, offset: 0, srvID: 379dd5e5-59c3-42cd-b9d3-67ca5e44560b, blockid: BP-617366194-10.1.0.252-1761586664431:blk_1073741913_1089, duration(ns): 486374
17:38:24.406 INFO DataNode - PacketResponder: BP-617366194-10.1.0.252-1761586664431:blk_1073741913_1089, type=LAST_IN_PIPELINE terminating
17:38:24.407 INFO BlockManagerInfo - Removed broadcast_505_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.9 MiB)
17:38:24.407 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest5_3a6038d6-4929-498a-89bd-0e9fb758a8e6.cram.parts/header is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:38:24.408 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest5_3a6038d6-4929-498a-89bd-0e9fb758a8e6.cram.parts/terminator dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:24.409 INFO StateChange - BLOCK* allocate blk_1073741914_1090, replicas=127.0.0.1:37927 for /user/runner/ReadsSparkSinkUnitTest5_3a6038d6-4929-498a-89bd-0e9fb758a8e6.cram.parts/terminator
17:38:24.409 INFO DataNode - Receiving BP-617366194-10.1.0.252-1761586664431:blk_1073741914_1090 src: /127.0.0.1:49806 dest: /127.0.0.1:37927
17:38:24.410 INFO clienttrace - src: /127.0.0.1:49806, dest: /127.0.0.1:37927, bytes: 38, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1369393814_1, offset: 0, srvID: 379dd5e5-59c3-42cd-b9d3-67ca5e44560b, blockid: BP-617366194-10.1.0.252-1761586664431:blk_1073741914_1090, duration(ns): 431004
17:38:24.410 INFO DataNode - PacketResponder: BP-617366194-10.1.0.252-1761586664431:blk_1073741914_1090, type=LAST_IN_PIPELINE terminating
17:38:24.411 INFO FSNamesystem - BLOCK* blk_1073741914_1090 is COMMITTED but not COMPLETE(numNodes= 0 < minimum = 1) in file /user/runner/ReadsSparkSinkUnitTest5_3a6038d6-4929-498a-89bd-0e9fb758a8e6.cram.parts/terminator
17:38:24.599 INFO FsDatasetAsyncDiskService - Scheduling blk_1073741906_1082 replica FinalizedReplica, blk_1073741906_1082, FINALIZED
getNumBytes() = 204
getBytesOnDisk() = 204
getVisibleLength()= 204
getVolume() = /tmp/minicluster_storage14473397491642437130/data/data2
getBlockURI() = file:/tmp/minicluster_storage14473397491642437130/data/data2/current/BP-617366194-10.1.0.252-1761586664431/current/finalized/subdir0/subdir0/blk_1073741906 for deletion
17:38:24.599 INFO FsDatasetAsyncDiskService - Scheduling blk_1073741907_1083 replica FinalizedReplica, blk_1073741907_1083, FINALIZED
getNumBytes() = 592
getBytesOnDisk() = 592
getVisibleLength()= 592
getVolume() = /tmp/minicluster_storage14473397491642437130/data/data1
getBlockURI() = file:/tmp/minicluster_storage14473397491642437130/data/data1/current/BP-617366194-10.1.0.252-1761586664431/current/finalized/subdir0/subdir0/blk_1073741907 for deletion
17:38:24.599 INFO FsDatasetAsyncDiskService - Scheduling blk_1073741899_1075 replica FinalizedReplica, blk_1073741899_1075, FINALIZED
getNumBytes() = 212
getBytesOnDisk() = 212
getVisibleLength()= 212
getVolume() = /tmp/minicluster_storage14473397491642437130/data/data1
getBlockURI() = file:/tmp/minicluster_storage14473397491642437130/data/data1/current/BP-617366194-10.1.0.252-1761586664431/current/finalized/subdir0/subdir0/blk_1073741899 for deletion
17:38:24.599 INFO FsDatasetAsyncDiskService - Deleted BP-617366194-10.1.0.252-1761586664431 blk_1073741906_1082 URI file:/tmp/minicluster_storage14473397491642437130/data/data2/current/BP-617366194-10.1.0.252-1761586664431/current/finalized/subdir0/subdir0/blk_1073741906
17:38:24.599 INFO FsDatasetAsyncDiskService - Scheduling blk_1073741900_1076 replica FinalizedReplica, blk_1073741900_1076, FINALIZED
getNumBytes() = 5472
getBytesOnDisk() = 5472
getVisibleLength()= 5472
getVolume() = /tmp/minicluster_storage14473397491642437130/data/data2
getBlockURI() = file:/tmp/minicluster_storage14473397491642437130/data/data2/current/BP-617366194-10.1.0.252-1761586664431/current/finalized/subdir0/subdir0/blk_1073741900 for deletion
17:38:24.599 INFO FsDatasetAsyncDiskService - Deleted BP-617366194-10.1.0.252-1761586664431 blk_1073741907_1083 URI file:/tmp/minicluster_storage14473397491642437130/data/data1/current/BP-617366194-10.1.0.252-1761586664431/current/finalized/subdir0/subdir0/blk_1073741907
17:38:24.599 INFO FsDatasetAsyncDiskService - Deleted BP-617366194-10.1.0.252-1761586664431 blk_1073741900_1076 URI file:/tmp/minicluster_storage14473397491642437130/data/data2/current/BP-617366194-10.1.0.252-1761586664431/current/finalized/subdir0/subdir0/blk_1073741900
17:38:24.600 INFO FsDatasetAsyncDiskService - Deleted BP-617366194-10.1.0.252-1761586664431 blk_1073741899_1075 URI file:/tmp/minicluster_storage14473397491642437130/data/data1/current/BP-617366194-10.1.0.252-1761586664431/current/finalized/subdir0/subdir0/blk_1073741899
17:38:24.812 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest5_3a6038d6-4929-498a-89bd-0e9fb758a8e6.cram.parts/terminator is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:38:24.813 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=listStatus src=/user/runner/ReadsSparkSinkUnitTest5_3a6038d6-4929-498a-89bd-0e9fb758a8e6.cram.parts dst=null perm=null proto=rpc
17:38:24.814 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest5_3a6038d6-4929-498a-89bd-0e9fb758a8e6.cram.parts/output dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:24.815 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest5_3a6038d6-4929-498a-89bd-0e9fb758a8e6.cram.parts/output is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:38:24.815 INFO HadoopFileSystemWrapper - Concatenating 3 parts to hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest5_3a6038d6-4929-498a-89bd-0e9fb758a8e6.cram
17:38:24.815 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=concat src=[/user/runner/ReadsSparkSinkUnitTest5_3a6038d6-4929-498a-89bd-0e9fb758a8e6.cram.parts/header, /user/runner/ReadsSparkSinkUnitTest5_3a6038d6-4929-498a-89bd-0e9fb758a8e6.cram.parts/part-r-00000, /user/runner/ReadsSparkSinkUnitTest5_3a6038d6-4929-498a-89bd-0e9fb758a8e6.cram.parts/terminator] dst=/user/runner/ReadsSparkSinkUnitTest5_3a6038d6-4929-498a-89bd-0e9fb758a8e6.cram.parts/output perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:24.816 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest5_3a6038d6-4929-498a-89bd-0e9fb758a8e6.cram dst=null perm=null proto=rpc
17:38:24.816 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=delete src=/user/runner/ReadsSparkSinkUnitTest5_3a6038d6-4929-498a-89bd-0e9fb758a8e6.cram dst=null perm=null proto=rpc
17:38:24.817 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=rename src=/user/runner/ReadsSparkSinkUnitTest5_3a6038d6-4929-498a-89bd-0e9fb758a8e6.cram.parts/output dst=/user/runner/ReadsSparkSinkUnitTest5_3a6038d6-4929-498a-89bd-0e9fb758a8e6.cram perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:24.817 INFO HadoopFileSystemWrapper - Concatenating to hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest5_3a6038d6-4929-498a-89bd-0e9fb758a8e6.cram done
17:38:24.817 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=delete src=/user/runner/ReadsSparkSinkUnitTest5_3a6038d6-4929-498a-89bd-0e9fb758a8e6.cram.parts dst=null perm=null proto=rpc
17:38:24.818 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest5_3a6038d6-4929-498a-89bd-0e9fb758a8e6.cram dst=null perm=null proto=rpc
17:38:24.818 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest5_3a6038d6-4929-498a-89bd-0e9fb758a8e6.cram dst=null perm=null proto=rpc
17:38:24.819 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest5_3a6038d6-4929-498a-89bd-0e9fb758a8e6.cram dst=null perm=null proto=rpc
17:38:24.819 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest5_3a6038d6-4929-498a-89bd-0e9fb758a8e6.cram dst=null perm=null proto=rpc
17:38:24.820 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest5_3a6038d6-4929-498a-89bd-0e9fb758a8e6.cram.crai dst=null perm=null proto=rpc
17:38:24.820 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest5_3a6038d6-4929-498a-89bd-0e9fb758a8e6.crai dst=null perm=null proto=rpc
17:38:24.823 WARN DFSUtil - Unexpected value for data transfer bytes=1024 duration=0
17:38:24.823 WARN DFSUtil - Unexpected value for data transfer bytes=42995 duration=0
17:38:24.824 WARN DFSUtil - Unexpected value for data transfer bytes=42 duration=0
17:38:24.824 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest5_3a6038d6-4929-498a-89bd-0e9fb758a8e6.cram dst=null perm=null proto=rpc
17:38:24.825 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest5_3a6038d6-4929-498a-89bd-0e9fb758a8e6.cram dst=null perm=null proto=rpc
17:38:24.825 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest5_3a6038d6-4929-498a-89bd-0e9fb758a8e6.cram.crai dst=null perm=null proto=rpc
17:38:24.825 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest5_3a6038d6-4929-498a-89bd-0e9fb758a8e6.crai dst=null perm=null proto=rpc
17:38:24.826 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest5_3a6038d6-4929-498a-89bd-0e9fb758a8e6.cram dst=null perm=null proto=rpc
17:38:24.826 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest5_3a6038d6-4929-498a-89bd-0e9fb758a8e6.cram dst=null perm=null proto=rpc
17:38:24.827 WARN DFSUtil - Unexpected value for data transfer bytes=1024 duration=0
17:38:24.828 WARN DFSUtil - Unexpected value for data transfer bytes=42995 duration=0
17:38:24.828 WARN DFSUtil - Unexpected value for data transfer bytes=42 duration=0
17:38:24.828 INFO MemoryStore - Block broadcast_510 stored as values in memory (estimated size 528.0 B, free 1919.6 MiB)
17:38:24.829 INFO MemoryStore - Block broadcast_510_piece0 stored as bytes in memory (estimated size 187.0 B, free 1919.6 MiB)
17:38:24.829 INFO BlockManagerInfo - Added broadcast_510_piece0 in memory on localhost:37953 (size: 187.0 B, free: 1919.9 MiB)
17:38:24.829 INFO SparkContext - Created broadcast 510 from broadcast at CramSource.java:114
17:38:24.830 INFO MemoryStore - Block broadcast_511 stored as values in memory (estimated size 297.9 KiB, free 1919.4 MiB)
17:38:24.836 INFO MemoryStore - Block broadcast_511_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1919.3 MiB)
17:38:24.836 INFO BlockManagerInfo - Added broadcast_511_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.9 MiB)
17:38:24.836 INFO SparkContext - Created broadcast 511 from newAPIHadoopFile at PathSplitSource.java:96
17:38:24.850 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest5_3a6038d6-4929-498a-89bd-0e9fb758a8e6.cram dst=null perm=null proto=rpc
17:38:24.850 INFO FileInputFormat - Total input files to process : 1
17:38:24.851 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest5_3a6038d6-4929-498a-89bd-0e9fb758a8e6.cram dst=null perm=null proto=rpc
17:38:24.874 INFO SparkContext - Starting job: collect at ReadsSparkSinkUnitTest.java:182
17:38:24.874 INFO DAGScheduler - Got job 190 (collect at ReadsSparkSinkUnitTest.java:182) with 1 output partitions
17:38:24.874 INFO DAGScheduler - Final stage: ResultStage 252 (collect at ReadsSparkSinkUnitTest.java:182)
17:38:24.874 INFO DAGScheduler - Parents of final stage: List()
17:38:24.874 INFO DAGScheduler - Missing parents: List()
17:38:24.874 INFO DAGScheduler - Submitting ResultStage 252 (MapPartitionsRDD[1214] at filter at ReadsSparkSource.java:96), which has no missing parents
17:38:24.885 INFO MemoryStore - Block broadcast_512 stored as values in memory (estimated size 286.8 KiB, free 1919.0 MiB)
17:38:24.886 INFO MemoryStore - Block broadcast_512_piece0 stored as bytes in memory (estimated size 103.6 KiB, free 1918.9 MiB)
17:38:24.886 INFO BlockManagerInfo - Added broadcast_512_piece0 in memory on localhost:37953 (size: 103.6 KiB, free: 1919.8 MiB)
17:38:24.886 INFO SparkContext - Created broadcast 512 from broadcast at DAGScheduler.scala:1580
17:38:24.886 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 252 (MapPartitionsRDD[1214] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:38:24.886 INFO TaskSchedulerImpl - Adding task set 252.0 with 1 tasks resource profile 0
17:38:24.887 INFO TaskSetManager - Starting task 0.0 in stage 252.0 (TID 308) (localhost, executor driver, partition 0, ANY, 7853 bytes)
17:38:24.887 INFO Executor - Running task 0.0 in stage 252.0 (TID 308)
17:38:24.909 INFO NewHadoopRDD - Input split: hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest5_3a6038d6-4929-498a-89bd-0e9fb758a8e6.cram:0+43713
17:38:24.910 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest5_3a6038d6-4929-498a-89bd-0e9fb758a8e6.cram dst=null perm=null proto=rpc
17:38:24.911 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest5_3a6038d6-4929-498a-89bd-0e9fb758a8e6.cram dst=null perm=null proto=rpc
17:38:24.911 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest5_3a6038d6-4929-498a-89bd-0e9fb758a8e6.cram.crai dst=null perm=null proto=rpc
17:38:24.912 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest5_3a6038d6-4929-498a-89bd-0e9fb758a8e6.crai dst=null perm=null proto=rpc
17:38:24.914 WARN DFSUtil - Unexpected value for data transfer bytes=1024 duration=0
17:38:24.915 WARN DFSUtil - Unexpected value for data transfer bytes=42995 duration=0
17:38:24.915 WARN DFSUtil - Unexpected value for data transfer bytes=42 duration=0
17:38:24.925 INFO Executor - Finished task 0.0 in stage 252.0 (TID 308). 154058 bytes result sent to driver
17:38:24.925 INFO TaskSetManager - Finished task 0.0 in stage 252.0 (TID 308) in 38 ms on localhost (executor driver) (1/1)
17:38:24.925 INFO TaskSchedulerImpl - Removed TaskSet 252.0, whose tasks have all completed, from pool
17:38:24.925 INFO DAGScheduler - ResultStage 252 (collect at ReadsSparkSinkUnitTest.java:182) finished in 0.050 s
17:38:24.925 INFO DAGScheduler - Job 190 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:24.926 INFO TaskSchedulerImpl - Killing all running tasks in stage 252: Stage finished
17:38:24.926 INFO DAGScheduler - Job 190 finished: collect at ReadsSparkSinkUnitTest.java:182, took 0.051945 s
17:38:24.931 INFO SparkContext - Starting job: count at ReadsSparkSinkUnitTest.java:185
17:38:24.931 INFO DAGScheduler - Got job 191 (count at ReadsSparkSinkUnitTest.java:185) with 1 output partitions
17:38:24.931 INFO DAGScheduler - Final stage: ResultStage 253 (count at ReadsSparkSinkUnitTest.java:185)
17:38:24.931 INFO DAGScheduler - Parents of final stage: List()
17:38:24.931 INFO DAGScheduler - Missing parents: List()
17:38:24.931 INFO DAGScheduler - Submitting ResultStage 253 (MapPartitionsRDD[1197] at filter at ReadsSparkSource.java:96), which has no missing parents
17:38:24.941 INFO MemoryStore - Block broadcast_513 stored as values in memory (estimated size 286.8 KiB, free 1918.6 MiB)
17:38:24.942 INFO MemoryStore - Block broadcast_513_piece0 stored as bytes in memory (estimated size 103.6 KiB, free 1918.5 MiB)
17:38:24.942 INFO BlockManagerInfo - Added broadcast_513_piece0 in memory on localhost:37953 (size: 103.6 KiB, free: 1919.7 MiB)
17:38:24.943 INFO SparkContext - Created broadcast 513 from broadcast at DAGScheduler.scala:1580
17:38:24.943 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 253 (MapPartitionsRDD[1197] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:38:24.943 INFO TaskSchedulerImpl - Adding task set 253.0 with 1 tasks resource profile 0
17:38:24.943 INFO TaskSetManager - Starting task 0.0 in stage 253.0 (TID 309) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7880 bytes)
17:38:24.943 INFO Executor - Running task 0.0 in stage 253.0 (TID 309)
17:38:24.965 INFO NewHadoopRDD - Input split: file:/home/runner/work/gatk/gatk/src/test/resources/org/broadinstitute/hellbender/tools/BQSR/NA12878.chr17_69k_70k.dictFix.cram:0+50619
17:38:24.971 INFO Executor - Finished task 0.0 in stage 253.0 (TID 309). 989 bytes result sent to driver
17:38:24.971 INFO TaskSetManager - Finished task 0.0 in stage 253.0 (TID 309) in 28 ms on localhost (executor driver) (1/1)
17:38:24.971 INFO TaskSchedulerImpl - Removed TaskSet 253.0, whose tasks have all completed, from pool
17:38:24.972 INFO DAGScheduler - ResultStage 253 (count at ReadsSparkSinkUnitTest.java:185) finished in 0.040 s
17:38:24.972 INFO DAGScheduler - Job 191 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:24.972 INFO TaskSchedulerImpl - Killing all running tasks in stage 253: Stage finished
17:38:24.972 INFO DAGScheduler - Job 191 finished: count at ReadsSparkSinkUnitTest.java:185, took 0.041181 s
17:38:24.975 INFO SparkContext - Starting job: count at ReadsSparkSinkUnitTest.java:185
17:38:24.975 INFO DAGScheduler - Got job 192 (count at ReadsSparkSinkUnitTest.java:185) with 1 output partitions
17:38:24.975 INFO DAGScheduler - Final stage: ResultStage 254 (count at ReadsSparkSinkUnitTest.java:185)
17:38:24.975 INFO DAGScheduler - Parents of final stage: List()
17:38:24.975 INFO DAGScheduler - Missing parents: List()
17:38:24.975 INFO DAGScheduler - Submitting ResultStage 254 (MapPartitionsRDD[1214] at filter at ReadsSparkSource.java:96), which has no missing parents
17:38:24.986 INFO MemoryStore - Block broadcast_514 stored as values in memory (estimated size 286.8 KiB, free 1918.3 MiB)
17:38:24.986 INFO MemoryStore - Block broadcast_514_piece0 stored as bytes in memory (estimated size 103.6 KiB, free 1918.2 MiB)
17:38:24.987 INFO BlockManagerInfo - Added broadcast_514_piece0 in memory on localhost:37953 (size: 103.6 KiB, free: 1919.6 MiB)
17:38:24.987 INFO SparkContext - Created broadcast 514 from broadcast at DAGScheduler.scala:1580
17:38:24.987 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 254 (MapPartitionsRDD[1214] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:38:24.987 INFO TaskSchedulerImpl - Adding task set 254.0 with 1 tasks resource profile 0
17:38:24.988 INFO TaskSetManager - Starting task 0.0 in stage 254.0 (TID 310) (localhost, executor driver, partition 0, ANY, 7853 bytes)
17:38:24.988 INFO Executor - Running task 0.0 in stage 254.0 (TID 310)
17:38:25.009 INFO NewHadoopRDD - Input split: hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest5_3a6038d6-4929-498a-89bd-0e9fb758a8e6.cram:0+43713
17:38:25.010 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest5_3a6038d6-4929-498a-89bd-0e9fb758a8e6.cram dst=null perm=null proto=rpc
17:38:25.010 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest5_3a6038d6-4929-498a-89bd-0e9fb758a8e6.cram dst=null perm=null proto=rpc
17:38:25.011 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest5_3a6038d6-4929-498a-89bd-0e9fb758a8e6.cram.crai dst=null perm=null proto=rpc
17:38:25.011 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest5_3a6038d6-4929-498a-89bd-0e9fb758a8e6.crai dst=null perm=null proto=rpc
17:38:25.014 WARN DFSUtil - Unexpected value for data transfer bytes=42995 duration=0
17:38:25.014 WARN DFSUtil - Unexpected value for data transfer bytes=42 duration=0
17:38:25.022 INFO Executor - Finished task 0.0 in stage 254.0 (TID 310). 989 bytes result sent to driver
17:38:25.023 INFO TaskSetManager - Finished task 0.0 in stage 254.0 (TID 310) in 36 ms on localhost (executor driver) (1/1)
17:38:25.023 INFO TaskSchedulerImpl - Removed TaskSet 254.0, whose tasks have all completed, from pool
17:38:25.023 INFO DAGScheduler - ResultStage 254 (count at ReadsSparkSinkUnitTest.java:185) finished in 0.048 s
17:38:25.023 INFO DAGScheduler - Job 192 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:25.023 INFO TaskSchedulerImpl - Killing all running tasks in stage 254: Stage finished
17:38:25.023 INFO DAGScheduler - Job 192 finished: count at ReadsSparkSinkUnitTest.java:185, took 0.048080 s
17:38:25.031 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest6_25958051-7338-45fe-b3f9-30ab4a8d7f8f.sam dst=null perm=null proto=rpc
17:38:25.032 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest6_25958051-7338-45fe-b3f9-30ab4a8d7f8f.sam dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:25.033 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest6_25958051-7338-45fe-b3f9-30ab4a8d7f8f.sam is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:38:25.033 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest6_25958051-7338-45fe-b3f9-30ab4a8d7f8f.sam dst=null perm=null proto=rpc
17:38:25.034 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:25.036 INFO MemoryStore - Block broadcast_515 stored as values in memory (estimated size 297.9 KiB, free 1917.9 MiB)
17:38:25.041 INFO MemoryStore - Block broadcast_515_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1917.8 MiB)
17:38:25.042 INFO BlockManagerInfo - Added broadcast_515_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.5 MiB)
17:38:25.042 INFO SparkContext - Created broadcast 515 from newAPIHadoopFile at PathSplitSource.java:96
17:38:25.061 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:25.063 INFO MemoryStore - Block broadcast_516 stored as values in memory (estimated size 297.9 KiB, free 1917.5 MiB)
17:38:25.068 INFO BlockManagerInfo - Removed broadcast_506_piece0 on localhost:37953 in memory (size: 1473.0 B, free: 1919.5 MiB)
17:38:25.069 INFO BlockManagerInfo - Removed broadcast_512_piece0 on localhost:37953 in memory (size: 103.6 KiB, free: 1919.6 MiB)
17:38:25.069 INFO BlockManagerInfo - Removed broadcast_503_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.7 MiB)
17:38:25.070 INFO BlockManagerInfo - Removed broadcast_511_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.7 MiB)
17:38:25.070 INFO BlockManagerInfo - Removed broadcast_513_piece0 on localhost:37953 in memory (size: 103.6 KiB, free: 1919.8 MiB)
17:38:25.071 INFO BlockManagerInfo - Removed broadcast_514_piece0 on localhost:37953 in memory (size: 103.6 KiB, free: 1919.9 MiB)
17:38:25.071 INFO MemoryStore - Block broadcast_516_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1919.3 MiB)
17:38:25.071 INFO BlockManagerInfo - Added broadcast_516_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.9 MiB)
17:38:25.072 INFO SparkContext - Created broadcast 516 from newAPIHadoopFile at PathSplitSource.java:96
17:38:25.072 INFO BlockManagerInfo - Removed broadcast_510_piece0 on localhost:37953 in memory (size: 187.0 B, free: 1919.9 MiB)
17:38:25.072 INFO BlockManagerInfo - Removed broadcast_502_piece0 on localhost:37953 in memory (size: 228.0 B, free: 1919.9 MiB)
17:38:25.073 INFO BlockManagerInfo - Removed broadcast_507_piece0 on localhost:37953 in memory (size: 1473.0 B, free: 1919.9 MiB)
17:38:25.091 INFO FileInputFormat - Total input files to process : 1
17:38:25.093 INFO MemoryStore - Block broadcast_517 stored as values in memory (estimated size 160.7 KiB, free 1919.2 MiB)
17:38:25.093 INFO MemoryStore - Block broadcast_517_piece0 stored as bytes in memory (estimated size 9.6 KiB, free 1919.2 MiB)
17:38:25.094 INFO BlockManagerInfo - Added broadcast_517_piece0 in memory on localhost:37953 (size: 9.6 KiB, free: 1919.9 MiB)
17:38:25.094 INFO SparkContext - Created broadcast 517 from broadcast at ReadsSparkSink.java:133
17:38:25.098 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest6_25958051-7338-45fe-b3f9-30ab4a8d7f8f.sam.parts dst=null perm=null proto=rpc
17:38:25.099 INFO HadoopMapRedCommitProtocol - Using output committer class org.apache.hadoop.mapred.FileOutputCommitter
17:38:25.099 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:25.099 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:25.099 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=mkdirs src=/user/runner/ReadsSparkSinkUnitTest6_25958051-7338-45fe-b3f9-30ab4a8d7f8f.sam.parts/_temporary/0 dst=null perm=runner:supergroup:rwxr-xr-x proto=rpc
17:38:25.105 INFO SparkContext - Starting job: runJob at SparkHadoopWriter.scala:83
17:38:25.106 INFO DAGScheduler - Registering RDD 1228 (mapToPair at SparkUtils.java:161) as input to shuffle 51
17:38:25.106 INFO DAGScheduler - Got job 193 (runJob at SparkHadoopWriter.scala:83) with 1 output partitions
17:38:25.106 INFO DAGScheduler - Final stage: ResultStage 256 (runJob at SparkHadoopWriter.scala:83)
17:38:25.106 INFO DAGScheduler - Parents of final stage: List(ShuffleMapStage 255)
17:38:25.106 INFO DAGScheduler - Missing parents: List(ShuffleMapStage 255)
17:38:25.106 INFO DAGScheduler - Submitting ShuffleMapStage 255 (MapPartitionsRDD[1228] at mapToPair at SparkUtils.java:161), which has no missing parents
17:38:25.121 INFO MemoryStore - Block broadcast_518 stored as values in memory (estimated size 520.4 KiB, free 1918.6 MiB)
17:38:25.123 INFO MemoryStore - Block broadcast_518_piece0 stored as bytes in memory (estimated size 166.1 KiB, free 1918.5 MiB)
17:38:25.123 INFO BlockManagerInfo - Added broadcast_518_piece0 in memory on localhost:37953 (size: 166.1 KiB, free: 1919.7 MiB)
17:38:25.123 INFO SparkContext - Created broadcast 518 from broadcast at DAGScheduler.scala:1580
17:38:25.123 INFO DAGScheduler - Submitting 1 missing tasks from ShuffleMapStage 255 (MapPartitionsRDD[1228] at mapToPair at SparkUtils.java:161) (first 15 tasks are for partitions Vector(0))
17:38:25.123 INFO TaskSchedulerImpl - Adding task set 255.0 with 1 tasks resource profile 0
17:38:25.124 INFO TaskSetManager - Starting task 0.0 in stage 255.0 (TID 311) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7861 bytes)
17:38:25.124 INFO Executor - Running task 0.0 in stage 255.0 (TID 311)
17:38:25.156 INFO NewHadoopRDD - Input split: file:/home/runner/work/gatk/gatk/src/test/resources/org/broadinstitute/hellbender/tools/BQSR/HiSeq.1mb.1RG.2k_lines.bam:0+222075
17:38:25.157 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:25.158 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:25.165 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:25.173 INFO Executor - Finished task 0.0 in stage 255.0 (TID 311). 1148 bytes result sent to driver
17:38:25.173 INFO TaskSetManager - Finished task 0.0 in stage 255.0 (TID 311) in 50 ms on localhost (executor driver) (1/1)
17:38:25.173 INFO TaskSchedulerImpl - Removed TaskSet 255.0, whose tasks have all completed, from pool
17:38:25.173 INFO DAGScheduler - ShuffleMapStage 255 (mapToPair at SparkUtils.java:161) finished in 0.067 s
17:38:25.173 INFO DAGScheduler - looking for newly runnable stages
17:38:25.173 INFO DAGScheduler - running: HashSet()
17:38:25.173 INFO DAGScheduler - waiting: HashSet(ResultStage 256)
17:38:25.173 INFO DAGScheduler - failed: HashSet()
17:38:25.174 INFO DAGScheduler - Submitting ResultStage 256 (MapPartitionsRDD[1234] at saveAsTextFile at SamSink.java:65), which has no missing parents
17:38:25.180 INFO MemoryStore - Block broadcast_519 stored as values in memory (estimated size 241.1 KiB, free 1918.2 MiB)
17:38:25.180 INFO MemoryStore - Block broadcast_519_piece0 stored as bytes in memory (estimated size 67.0 KiB, free 1918.2 MiB)
17:38:25.180 INFO BlockManagerInfo - Added broadcast_519_piece0 in memory on localhost:37953 (size: 67.0 KiB, free: 1919.7 MiB)
17:38:25.181 INFO SparkContext - Created broadcast 519 from broadcast at DAGScheduler.scala:1580
17:38:25.181 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 256 (MapPartitionsRDD[1234] at saveAsTextFile at SamSink.java:65) (first 15 tasks are for partitions Vector(0))
17:38:25.181 INFO TaskSchedulerImpl - Adding task set 256.0 with 1 tasks resource profile 0
17:38:25.181 INFO TaskSetManager - Starting task 0.0 in stage 256.0 (TID 312) (localhost, executor driver, partition 0, NODE_LOCAL, 7513 bytes)
17:38:25.182 INFO Executor - Running task 0.0 in stage 256.0 (TID 312)
17:38:25.186 INFO ShuffleBlockFetcherIterator - Getting 1 (343.8 KiB) non-empty blocks including 1 (343.8 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
17:38:25.186 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
17:38:25.197 INFO HadoopMapRedCommitProtocol - Using output committer class org.apache.hadoop.mapred.FileOutputCommitter
17:38:25.197 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:25.197 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:25.198 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest6_25958051-7338-45fe-b3f9-30ab4a8d7f8f.sam.parts/_temporary/0/_temporary/attempt_202510271738252584518704520081199_1234_m_000000_0/part-00000 dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:25.200 INFO StateChange - BLOCK* allocate blk_1073741915_1091, replicas=127.0.0.1:37927 for /user/runner/ReadsSparkSinkUnitTest6_25958051-7338-45fe-b3f9-30ab4a8d7f8f.sam.parts/_temporary/0/_temporary/attempt_202510271738252584518704520081199_1234_m_000000_0/part-00000
17:38:25.201 INFO DataNode - Receiving BP-617366194-10.1.0.252-1761586664431:blk_1073741915_1091 src: /127.0.0.1:49812 dest: /127.0.0.1:37927
17:38:25.206 INFO clienttrace - src: /127.0.0.1:49812, dest: /127.0.0.1:37927, bytes: 761729, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1369393814_1, offset: 0, srvID: 379dd5e5-59c3-42cd-b9d3-67ca5e44560b, blockid: BP-617366194-10.1.0.252-1761586664431:blk_1073741915_1091, duration(ns): 4279780
17:38:25.206 INFO DataNode - PacketResponder: BP-617366194-10.1.0.252-1761586664431:blk_1073741915_1091, type=LAST_IN_PIPELINE terminating
17:38:25.207 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest6_25958051-7338-45fe-b3f9-30ab4a8d7f8f.sam.parts/_temporary/0/_temporary/attempt_202510271738252584518704520081199_1234_m_000000_0/part-00000 is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:38:25.208 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest6_25958051-7338-45fe-b3f9-30ab4a8d7f8f.sam.parts/_temporary/0/_temporary/attempt_202510271738252584518704520081199_1234_m_000000_0 dst=null perm=null proto=rpc
17:38:25.208 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest6_25958051-7338-45fe-b3f9-30ab4a8d7f8f.sam.parts/_temporary/0/_temporary/attempt_202510271738252584518704520081199_1234_m_000000_0 dst=null perm=null proto=rpc
17:38:25.209 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest6_25958051-7338-45fe-b3f9-30ab4a8d7f8f.sam.parts/_temporary/0/task_202510271738252584518704520081199_1234_m_000000 dst=null perm=null proto=rpc
17:38:25.209 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=rename src=/user/runner/ReadsSparkSinkUnitTest6_25958051-7338-45fe-b3f9-30ab4a8d7f8f.sam.parts/_temporary/0/_temporary/attempt_202510271738252584518704520081199_1234_m_000000_0 dst=/user/runner/ReadsSparkSinkUnitTest6_25958051-7338-45fe-b3f9-30ab4a8d7f8f.sam.parts/_temporary/0/task_202510271738252584518704520081199_1234_m_000000 perm=runner:supergroup:rwxr-xr-x proto=rpc
17:38:25.209 INFO FileOutputCommitter - Saved output of task 'attempt_202510271738252584518704520081199_1234_m_000000_0' to hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest6_25958051-7338-45fe-b3f9-30ab4a8d7f8f.sam.parts/_temporary/0/task_202510271738252584518704520081199_1234_m_000000
17:38:25.209 INFO SparkHadoopMapRedUtil - attempt_202510271738252584518704520081199_1234_m_000000_0: Committed. Elapsed time: 1 ms.
17:38:25.210 INFO Executor - Finished task 0.0 in stage 256.0 (TID 312). 1858 bytes result sent to driver
17:38:25.210 INFO TaskSetManager - Finished task 0.0 in stage 256.0 (TID 312) in 29 ms on localhost (executor driver) (1/1)
17:38:25.210 INFO TaskSchedulerImpl - Removed TaskSet 256.0, whose tasks have all completed, from pool
17:38:25.210 INFO DAGScheduler - ResultStage 256 (runJob at SparkHadoopWriter.scala:83) finished in 0.036 s
17:38:25.211 INFO DAGScheduler - Job 193 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:25.211 INFO TaskSchedulerImpl - Killing all running tasks in stage 256: Stage finished
17:38:25.211 INFO DAGScheduler - Job 193 finished: runJob at SparkHadoopWriter.scala:83, took 0.105829 s
17:38:25.211 INFO SparkHadoopWriter - Start to commit write Job job_202510271738252584518704520081199_1234.
17:38:25.212 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=listStatus src=/user/runner/ReadsSparkSinkUnitTest6_25958051-7338-45fe-b3f9-30ab4a8d7f8f.sam.parts/_temporary/0 dst=null perm=null proto=rpc
17:38:25.213 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest6_25958051-7338-45fe-b3f9-30ab4a8d7f8f.sam.parts dst=null perm=null proto=rpc
17:38:25.213 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=listStatus src=/user/runner/ReadsSparkSinkUnitTest6_25958051-7338-45fe-b3f9-30ab4a8d7f8f.sam.parts/_temporary/0/task_202510271738252584518704520081199_1234_m_000000 dst=null perm=null proto=rpc
17:38:25.213 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest6_25958051-7338-45fe-b3f9-30ab4a8d7f8f.sam.parts/part-00000 dst=null perm=null proto=rpc
17:38:25.214 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=rename src=/user/runner/ReadsSparkSinkUnitTest6_25958051-7338-45fe-b3f9-30ab4a8d7f8f.sam.parts/_temporary/0/task_202510271738252584518704520081199_1234_m_000000/part-00000 dst=/user/runner/ReadsSparkSinkUnitTest6_25958051-7338-45fe-b3f9-30ab4a8d7f8f.sam.parts/part-00000 perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:25.215 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=delete src=/user/runner/ReadsSparkSinkUnitTest6_25958051-7338-45fe-b3f9-30ab4a8d7f8f.sam.parts/_temporary dst=null perm=null proto=rpc
17:38:25.215 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest6_25958051-7338-45fe-b3f9-30ab4a8d7f8f.sam.parts/_SUCCESS dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:25.216 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest6_25958051-7338-45fe-b3f9-30ab4a8d7f8f.sam.parts/_SUCCESS is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:38:25.217 INFO audit - allowed=false ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=delete src=/user/runner/ReadsSparkSinkUnitTest6_25958051-7338-45fe-b3f9-30ab4a8d7f8f.sam.parts/.spark-staging-1234 dst=null perm=null proto=rpc
17:38:25.217 INFO SparkHadoopWriter - Write Job job_202510271738252584518704520081199_1234 committed. Elapsed time: 5 ms.
17:38:25.217 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest6_25958051-7338-45fe-b3f9-30ab4a8d7f8f.sam.parts/header dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:25.218 INFO StateChange - BLOCK* allocate blk_1073741916_1092, replicas=127.0.0.1:37927 for /user/runner/ReadsSparkSinkUnitTest6_25958051-7338-45fe-b3f9-30ab4a8d7f8f.sam.parts/header
17:38:25.219 INFO DataNode - Receiving BP-617366194-10.1.0.252-1761586664431:blk_1073741916_1092 src: /127.0.0.1:49820 dest: /127.0.0.1:37927
17:38:25.220 INFO clienttrace - src: /127.0.0.1:49820, dest: /127.0.0.1:37927, bytes: 85829, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1369393814_1, offset: 0, srvID: 379dd5e5-59c3-42cd-b9d3-67ca5e44560b, blockid: BP-617366194-10.1.0.252-1761586664431:blk_1073741916_1092, duration(ns): 516383
17:38:25.220 INFO DataNode - PacketResponder: BP-617366194-10.1.0.252-1761586664431:blk_1073741916_1092, type=LAST_IN_PIPELINE terminating
17:38:25.221 INFO FSNamesystem - BLOCK* blk_1073741916_1092 is COMMITTED but not COMPLETE(numNodes= 0 < minimum = 1) in file /user/runner/ReadsSparkSinkUnitTest6_25958051-7338-45fe-b3f9-30ab4a8d7f8f.sam.parts/header
17:38:25.622 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest6_25958051-7338-45fe-b3f9-30ab4a8d7f8f.sam.parts/header is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:38:25.622 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=listStatus src=/user/runner/ReadsSparkSinkUnitTest6_25958051-7338-45fe-b3f9-30ab4a8d7f8f.sam.parts dst=null perm=null proto=rpc
17:38:25.623 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=create src=/user/runner/ReadsSparkSinkUnitTest6_25958051-7338-45fe-b3f9-30ab4a8d7f8f.sam.parts/output dst=null perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:25.624 INFO StateChange - DIR* completeFile: /user/runner/ReadsSparkSinkUnitTest6_25958051-7338-45fe-b3f9-30ab4a8d7f8f.sam.parts/output is closed by DFSClient_NONMAPREDUCE_-1369393814_1
17:38:25.624 INFO HadoopFileSystemWrapper - Concatenating 2 parts to hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest6_25958051-7338-45fe-b3f9-30ab4a8d7f8f.sam
17:38:25.624 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=concat src=[/user/runner/ReadsSparkSinkUnitTest6_25958051-7338-45fe-b3f9-30ab4a8d7f8f.sam.parts/header, /user/runner/ReadsSparkSinkUnitTest6_25958051-7338-45fe-b3f9-30ab4a8d7f8f.sam.parts/part-00000] dst=/user/runner/ReadsSparkSinkUnitTest6_25958051-7338-45fe-b3f9-30ab4a8d7f8f.sam.parts/output perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:25.625 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest6_25958051-7338-45fe-b3f9-30ab4a8d7f8f.sam dst=null perm=null proto=rpc
17:38:25.625 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=delete src=/user/runner/ReadsSparkSinkUnitTest6_25958051-7338-45fe-b3f9-30ab4a8d7f8f.sam dst=null perm=null proto=rpc
17:38:25.626 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=rename src=/user/runner/ReadsSparkSinkUnitTest6_25958051-7338-45fe-b3f9-30ab4a8d7f8f.sam.parts/output dst=/user/runner/ReadsSparkSinkUnitTest6_25958051-7338-45fe-b3f9-30ab4a8d7f8f.sam perm=runner:supergroup:rw-r--r-- proto=rpc
17:38:25.626 INFO HadoopFileSystemWrapper - Concatenating to hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest6_25958051-7338-45fe-b3f9-30ab4a8d7f8f.sam done
17:38:25.627 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=delete src=/user/runner/ReadsSparkSinkUnitTest6_25958051-7338-45fe-b3f9-30ab4a8d7f8f.sam.parts dst=null perm=null proto=rpc
17:38:25.627 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest6_25958051-7338-45fe-b3f9-30ab4a8d7f8f.sam dst=null perm=null proto=rpc
17:38:25.628 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest6_25958051-7338-45fe-b3f9-30ab4a8d7f8f.sam dst=null perm=null proto=rpc
17:38:25.628 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest6_25958051-7338-45fe-b3f9-30ab4a8d7f8f.sam dst=null perm=null proto=rpc
17:38:25.629 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest6_25958051-7338-45fe-b3f9-30ab4a8d7f8f.sam dst=null perm=null proto=rpc
WARNING 2025-10-27 17:38:25 SamReaderFactory Unable to detect file format from input URL or stream, assuming SAM format.
17:38:25.631 WARN DFSUtil - Unexpected value for data transfer bytes=86501 duration=0
17:38:25.632 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest6_25958051-7338-45fe-b3f9-30ab4a8d7f8f.sam dst=null perm=null proto=rpc
17:38:25.633 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest6_25958051-7338-45fe-b3f9-30ab4a8d7f8f.sam dst=null perm=null proto=rpc
17:38:25.633 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest6_25958051-7338-45fe-b3f9-30ab4a8d7f8f.sam dst=null perm=null proto=rpc
WARNING 2025-10-27 17:38:25 SamReaderFactory Unable to detect file format from input URL or stream, assuming SAM format.
17:38:25.635 WARN DFSUtil - Unexpected value for data transfer bytes=86501 duration=0
17:38:25.636 WARN DFSUtil - Unexpected value for data transfer bytes=767681 duration=0
17:38:25.636 INFO MemoryStore - Block broadcast_520 stored as values in memory (estimated size 160.7 KiB, free 1918.0 MiB)
17:38:25.637 INFO MemoryStore - Block broadcast_520_piece0 stored as bytes in memory (estimated size 9.6 KiB, free 1918.0 MiB)
17:38:25.637 INFO BlockManagerInfo - Added broadcast_520_piece0 in memory on localhost:37953 (size: 9.6 KiB, free: 1919.7 MiB)
17:38:25.639 INFO SparkContext - Created broadcast 520 from broadcast at SamSource.java:78
17:38:25.640 INFO MemoryStore - Block broadcast_521 stored as values in memory (estimated size 297.9 KiB, free 1917.7 MiB)
17:38:25.645 INFO MemoryStore - Block broadcast_521_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1917.7 MiB)
17:38:25.645 INFO BlockManagerInfo - Added broadcast_521_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.6 MiB)
17:38:25.645 INFO SparkContext - Created broadcast 521 from newAPIHadoopFile at SamSource.java:108
17:38:25.649 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/user/runner/ReadsSparkSinkUnitTest6_25958051-7338-45fe-b3f9-30ab4a8d7f8f.sam dst=null perm=null proto=rpc
17:38:25.649 INFO FileInputFormat - Total input files to process : 1
17:38:25.649 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest6_25958051-7338-45fe-b3f9-30ab4a8d7f8f.sam dst=null perm=null proto=rpc
17:38:25.653 INFO SparkContext - Starting job: collect at ReadsSparkSinkUnitTest.java:182
17:38:25.654 INFO DAGScheduler - Got job 194 (collect at ReadsSparkSinkUnitTest.java:182) with 1 output partitions
17:38:25.654 INFO DAGScheduler - Final stage: ResultStage 257 (collect at ReadsSparkSinkUnitTest.java:182)
17:38:25.654 INFO DAGScheduler - Parents of final stage: List()
17:38:25.654 INFO DAGScheduler - Missing parents: List()
17:38:25.654 INFO DAGScheduler - Submitting ResultStage 257 (MapPartitionsRDD[1239] at filter at ReadsSparkSource.java:96), which has no missing parents
17:38:25.655 INFO MemoryStore - Block broadcast_522 stored as values in memory (estimated size 7.5 KiB, free 1917.7 MiB)
17:38:25.658 INFO MemoryStore - Block broadcast_522_piece0 stored as bytes in memory (estimated size 3.8 KiB, free 1917.7 MiB)
17:38:25.658 INFO BlockManagerInfo - Added broadcast_522_piece0 in memory on localhost:37953 (size: 3.8 KiB, free: 1919.6 MiB)
17:38:25.658 INFO SparkContext - Created broadcast 522 from broadcast at DAGScheduler.scala:1580
17:38:25.658 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 257 (MapPartitionsRDD[1239] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:38:25.659 INFO TaskSchedulerImpl - Adding task set 257.0 with 1 tasks resource profile 0
17:38:25.659 INFO BlockManagerInfo - Removed broadcast_516_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.7 MiB)
17:38:25.659 INFO TaskSetManager - Starting task 0.0 in stage 257.0 (TID 313) (localhost, executor driver, partition 0, ANY, 7852 bytes)
17:38:25.659 INFO Executor - Running task 0.0 in stage 257.0 (TID 313)
17:38:25.660 INFO BlockManagerInfo - Removed broadcast_518_piece0 on localhost:37953 in memory (size: 166.1 KiB, free: 1919.8 MiB)
17:38:25.660 INFO BlockManagerInfo - Removed broadcast_517_piece0 on localhost:37953 in memory (size: 9.6 KiB, free: 1919.8 MiB)
17:38:25.660 INFO BlockManagerInfo - Removed broadcast_519_piece0 on localhost:37953 in memory (size: 67.0 KiB, free: 1919.9 MiB)
17:38:25.661 INFO NewHadoopRDD - Input split: hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest6_25958051-7338-45fe-b3f9-30ab4a8d7f8f.sam:0+847558
17:38:25.662 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest6_25958051-7338-45fe-b3f9-30ab4a8d7f8f.sam dst=null perm=null proto=rpc
17:38:25.664 WARN DFSUtil - Unexpected value for data transfer bytes=86501 duration=0
17:38:25.672 INFO Executor - Finished task 0.0 in stage 257.0 (TID 313). 651526 bytes result sent to driver
17:38:25.674 INFO TaskSetManager - Finished task 0.0 in stage 257.0 (TID 313) in 14 ms on localhost (executor driver) (1/1)
17:38:25.674 INFO TaskSchedulerImpl - Removed TaskSet 257.0, whose tasks have all completed, from pool
17:38:25.674 INFO DAGScheduler - ResultStage 257 (collect at ReadsSparkSinkUnitTest.java:182) finished in 0.020 s
17:38:25.674 INFO DAGScheduler - Job 194 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:25.674 INFO TaskSchedulerImpl - Killing all running tasks in stage 257: Stage finished
17:38:25.674 INFO DAGScheduler - Job 194 finished: collect at ReadsSparkSinkUnitTest.java:182, took 0.020360 s
17:38:25.684 INFO SparkContext - Starting job: count at ReadsSparkSinkUnitTest.java:185
17:38:25.684 INFO DAGScheduler - Got job 195 (count at ReadsSparkSinkUnitTest.java:185) with 1 output partitions
17:38:25.684 INFO DAGScheduler - Final stage: ResultStage 258 (count at ReadsSparkSinkUnitTest.java:185)
17:38:25.684 INFO DAGScheduler - Parents of final stage: List()
17:38:25.684 INFO DAGScheduler - Missing parents: List()
17:38:25.684 INFO DAGScheduler - Submitting ResultStage 258 (MapPartitionsRDD[1221] at filter at ReadsSparkSource.java:96), which has no missing parents
17:38:25.699 INFO MemoryStore - Block broadcast_523 stored as values in memory (estimated size 426.1 KiB, free 1918.7 MiB)
17:38:25.700 INFO MemoryStore - Block broadcast_523_piece0 stored as bytes in memory (estimated size 153.6 KiB, free 1918.6 MiB)
17:38:25.700 INFO BlockManagerInfo - Added broadcast_523_piece0 in memory on localhost:37953 (size: 153.6 KiB, free: 1919.7 MiB)
17:38:25.700 INFO SparkContext - Created broadcast 523 from broadcast at DAGScheduler.scala:1580
17:38:25.701 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 258 (MapPartitionsRDD[1221] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:38:25.701 INFO TaskSchedulerImpl - Adding task set 258.0 with 1 tasks resource profile 0
17:38:25.701 INFO TaskSetManager - Starting task 0.0 in stage 258.0 (TID 314) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7872 bytes)
17:38:25.701 INFO Executor - Running task 0.0 in stage 258.0 (TID 314)
17:38:25.733 INFO NewHadoopRDD - Input split: file:/home/runner/work/gatk/gatk/src/test/resources/org/broadinstitute/hellbender/tools/BQSR/HiSeq.1mb.1RG.2k_lines.bam:0+222075
17:38:25.733 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:25.734 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:25.741 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:25.745 INFO Executor - Finished task 0.0 in stage 258.0 (TID 314). 989 bytes result sent to driver
17:38:25.745 INFO TaskSetManager - Finished task 0.0 in stage 258.0 (TID 314) in 44 ms on localhost (executor driver) (1/1)
17:38:25.745 INFO TaskSchedulerImpl - Removed TaskSet 258.0, whose tasks have all completed, from pool
17:38:25.745 INFO DAGScheduler - ResultStage 258 (count at ReadsSparkSinkUnitTest.java:185) finished in 0.061 s
17:38:25.745 INFO DAGScheduler - Job 195 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:25.745 INFO TaskSchedulerImpl - Killing all running tasks in stage 258: Stage finished
17:38:25.745 INFO DAGScheduler - Job 195 finished: count at ReadsSparkSinkUnitTest.java:185, took 0.061777 s
17:38:25.748 INFO SparkContext - Starting job: count at ReadsSparkSinkUnitTest.java:185
17:38:25.749 INFO DAGScheduler - Got job 196 (count at ReadsSparkSinkUnitTest.java:185) with 1 output partitions
17:38:25.749 INFO DAGScheduler - Final stage: ResultStage 259 (count at ReadsSparkSinkUnitTest.java:185)
17:38:25.749 INFO DAGScheduler - Parents of final stage: List()
17:38:25.749 INFO DAGScheduler - Missing parents: List()
17:38:25.749 INFO DAGScheduler - Submitting ResultStage 259 (MapPartitionsRDD[1239] at filter at ReadsSparkSource.java:96), which has no missing parents
17:38:25.749 INFO MemoryStore - Block broadcast_524 stored as values in memory (estimated size 7.4 KiB, free 1918.6 MiB)
17:38:25.750 INFO MemoryStore - Block broadcast_524_piece0 stored as bytes in memory (estimated size 3.8 KiB, free 1918.6 MiB)
17:38:25.750 INFO BlockManagerInfo - Added broadcast_524_piece0 in memory on localhost:37953 (size: 3.8 KiB, free: 1919.7 MiB)
17:38:25.750 INFO SparkContext - Created broadcast 524 from broadcast at DAGScheduler.scala:1580
17:38:25.750 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 259 (MapPartitionsRDD[1239] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:38:25.750 INFO TaskSchedulerImpl - Adding task set 259.0 with 1 tasks resource profile 0
17:38:25.751 INFO TaskSetManager - Starting task 0.0 in stage 259.0 (TID 315) (localhost, executor driver, partition 0, ANY, 7852 bytes)
17:38:25.751 INFO Executor - Running task 0.0 in stage 259.0 (TID 315)
17:38:25.752 INFO NewHadoopRDD - Input split: hdfs://localhost:38243/user/runner/ReadsSparkSinkUnitTest6_25958051-7338-45fe-b3f9-30ab4a8d7f8f.sam:0+847558
17:38:25.753 INFO audit - allowed=true ugi=runner (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/user/runner/ReadsSparkSinkUnitTest6_25958051-7338-45fe-b3f9-30ab4a8d7f8f.sam dst=null perm=null proto=rpc
17:38:25.754 WARN DFSUtil - Unexpected value for data transfer bytes=86501 duration=0
17:38:25.759 WARN DFSUtil - Unexpected value for data transfer bytes=767681 duration=0
17:38:25.760 INFO Executor - Finished task 0.0 in stage 259.0 (TID 315). 946 bytes result sent to driver
17:38:25.760 INFO TaskSetManager - Finished task 0.0 in stage 259.0 (TID 315) in 10 ms on localhost (executor driver) (1/1)
17:38:25.760 INFO TaskSchedulerImpl - Removed TaskSet 259.0, whose tasks have all completed, from pool
17:38:25.760 INFO DAGScheduler - ResultStage 259 (count at ReadsSparkSinkUnitTest.java:185) finished in 0.011 s
17:38:25.760 INFO DAGScheduler - Job 196 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:25.760 INFO TaskSchedulerImpl - Killing all running tasks in stage 259: Stage finished
17:38:25.760 INFO DAGScheduler - Job 196 finished: count at ReadsSparkSinkUnitTest.java:185, took 0.011844 s
17:38:25.761 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:25.763 INFO MemoryStore - Block broadcast_525 stored as values in memory (estimated size 297.9 KiB, free 1918.3 MiB)
17:38:25.769 INFO MemoryStore - Block broadcast_525_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1918.2 MiB)
17:38:25.769 INFO BlockManagerInfo - Added broadcast_525_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.7 MiB)
17:38:25.769 INFO SparkContext - Created broadcast 525 from newAPIHadoopFile at PathSplitSource.java:96
17:38:25.788 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:25.790 INFO MemoryStore - Block broadcast_526 stored as values in memory (estimated size 297.9 KiB, free 1917.9 MiB)
17:38:25.796 INFO MemoryStore - Block broadcast_526_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1917.9 MiB)
17:38:25.796 INFO BlockManagerInfo - Added broadcast_526_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.6 MiB)
17:38:25.796 INFO SparkContext - Created broadcast 526 from newAPIHadoopFile at PathSplitSource.java:96
17:38:25.815 INFO FileInputFormat - Total input files to process : 1
17:38:25.817 INFO MemoryStore - Block broadcast_527 stored as values in memory (estimated size 160.7 KiB, free 1917.7 MiB)
17:38:25.817 INFO MemoryStore - Block broadcast_527_piece0 stored as bytes in memory (estimated size 9.6 KiB, free 1917.7 MiB)
17:38:25.817 INFO BlockManagerInfo - Added broadcast_527_piece0 in memory on localhost:37953 (size: 9.6 KiB, free: 1919.6 MiB)
17:38:25.818 INFO SparkContext - Created broadcast 527 from broadcast at ReadsSparkSink.java:133
17:38:25.819 INFO MemoryStore - Block broadcast_528 stored as values in memory (estimated size 163.2 KiB, free 1917.6 MiB)
17:38:25.823 INFO MemoryStore - Block broadcast_528_piece0 stored as bytes in memory (estimated size 9.6 KiB, free 1917.9 MiB)
17:38:25.823 INFO BlockManagerInfo - Removed broadcast_521_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.7 MiB)
17:38:25.823 INFO BlockManagerInfo - Added broadcast_528_piece0 in memory on localhost:37953 (size: 9.6 KiB, free: 1919.7 MiB)
17:38:25.823 INFO SparkContext - Created broadcast 528 from broadcast at BamSink.java:76
17:38:25.823 INFO BlockManagerInfo - Removed broadcast_520_piece0 on localhost:37953 in memory (size: 9.6 KiB, free: 1919.7 MiB)
17:38:25.823 INFO BlockManagerInfo - Removed broadcast_524_piece0 on localhost:37953 in memory (size: 3.8 KiB, free: 1919.7 MiB)
17:38:25.824 INFO BlockManagerInfo - Removed broadcast_515_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.7 MiB)
17:38:25.824 INFO BlockManagerInfo - Removed broadcast_522_piece0 on localhost:37953 in memory (size: 3.8 KiB, free: 1919.7 MiB)
17:38:25.825 INFO BlockManagerInfo - Removed broadcast_526_piece0 on localhost:37953 in memory (size: 50.2 KiB, free: 1919.8 MiB)
17:38:25.825 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:25.825 INFO BlockManagerInfo - Removed broadcast_523_piece0 on localhost:37953 in memory (size: 153.6 KiB, free: 1919.9 MiB)
17:38:25.825 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:25.825 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:25.838 INFO SparkContext - Starting job: runJob at SparkHadoopWriter.scala:83
17:38:25.838 INFO DAGScheduler - Registering RDD 1253 (mapToPair at SparkUtils.java:161) as input to shuffle 52
17:38:25.839 INFO DAGScheduler - Got job 197 (runJob at SparkHadoopWriter.scala:83) with 1 output partitions
17:38:25.839 INFO DAGScheduler - Final stage: ResultStage 261 (runJob at SparkHadoopWriter.scala:83)
17:38:25.839 INFO DAGScheduler - Parents of final stage: List(ShuffleMapStage 260)
17:38:25.839 INFO DAGScheduler - Missing parents: List(ShuffleMapStage 260)
17:38:25.839 INFO DAGScheduler - Submitting ShuffleMapStage 260 (MapPartitionsRDD[1253] at mapToPair at SparkUtils.java:161), which has no missing parents
17:38:25.855 INFO MemoryStore - Block broadcast_529 stored as values in memory (estimated size 520.4 KiB, free 1918.8 MiB)
17:38:25.856 INFO MemoryStore - Block broadcast_529_piece0 stored as bytes in memory (estimated size 166.1 KiB, free 1918.7 MiB)
17:38:25.856 INFO BlockManagerInfo - Added broadcast_529_piece0 in memory on localhost:37953 (size: 166.1 KiB, free: 1919.8 MiB)
17:38:25.856 INFO SparkContext - Created broadcast 529 from broadcast at DAGScheduler.scala:1580
17:38:25.856 INFO DAGScheduler - Submitting 1 missing tasks from ShuffleMapStage 260 (MapPartitionsRDD[1253] at mapToPair at SparkUtils.java:161) (first 15 tasks are for partitions Vector(0))
17:38:25.856 INFO TaskSchedulerImpl - Adding task set 260.0 with 1 tasks resource profile 0
17:38:25.857 INFO TaskSetManager - Starting task 0.0 in stage 260.0 (TID 316) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7861 bytes)
17:38:25.857 INFO Executor - Running task 0.0 in stage 260.0 (TID 316)
17:38:25.889 INFO NewHadoopRDD - Input split: file:/home/runner/work/gatk/gatk/src/test/resources/org/broadinstitute/hellbender/tools/BQSR/HiSeq.1mb.1RG.2k_lines.bam:0+222075
17:38:25.890 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:25.891 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:25.898 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:25.906 INFO Executor - Finished task 0.0 in stage 260.0 (TID 316). 1148 bytes result sent to driver
17:38:25.907 INFO TaskSetManager - Finished task 0.0 in stage 260.0 (TID 316) in 50 ms on localhost (executor driver) (1/1)
17:38:25.907 INFO TaskSchedulerImpl - Removed TaskSet 260.0, whose tasks have all completed, from pool
17:38:25.907 INFO DAGScheduler - ShuffleMapStage 260 (mapToPair at SparkUtils.java:161) finished in 0.068 s
17:38:25.907 INFO DAGScheduler - looking for newly runnable stages
17:38:25.907 INFO DAGScheduler - running: HashSet()
17:38:25.907 INFO DAGScheduler - waiting: HashSet(ResultStage 261)
17:38:25.907 INFO DAGScheduler - failed: HashSet()
17:38:25.907 INFO DAGScheduler - Submitting ResultStage 261 (MapPartitionsRDD[1258] at mapToPair at BamSink.java:91), which has no missing parents
17:38:25.913 INFO MemoryStore - Block broadcast_530 stored as values in memory (estimated size 241.4 KiB, free 1918.4 MiB)
17:38:25.914 INFO MemoryStore - Block broadcast_530_piece0 stored as bytes in memory (estimated size 66.9 KiB, free 1918.4 MiB)
17:38:25.914 INFO BlockManagerInfo - Added broadcast_530_piece0 in memory on localhost:37953 (size: 66.9 KiB, free: 1919.7 MiB)
17:38:25.914 INFO SparkContext - Created broadcast 530 from broadcast at DAGScheduler.scala:1580
17:38:25.914 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 261 (MapPartitionsRDD[1258] at mapToPair at BamSink.java:91) (first 15 tasks are for partitions Vector(0))
17:38:25.914 INFO TaskSchedulerImpl - Adding task set 261.0 with 1 tasks resource profile 0
17:38:25.915 INFO TaskSetManager - Starting task 0.0 in stage 261.0 (TID 317) (localhost, executor driver, partition 0, NODE_LOCAL, 7513 bytes)
17:38:25.915 INFO Executor - Running task 0.0 in stage 261.0 (TID 317)
17:38:25.920 INFO ShuffleBlockFetcherIterator - Getting 1 (343.8 KiB) non-empty blocks including 1 (343.8 KiB) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
17:38:25.920 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
17:38:25.931 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:25.931 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:25.931 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:25.931 INFO PathOutputCommitterFactory - No output committer factory defined, defaulting to FileOutputCommitterFactory
17:38:25.931 INFO FileOutputCommitter - File Output Committer Algorithm version is 1
17:38:25.931 INFO FileOutputCommitter - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
17:38:25.939 WARN IntelDeflaterFactory - Intel Deflater not supported, using Java.util.zip.Deflater
17:38:25.955 INFO FileOutputCommitter - Saved output of task 'attempt_202510271738254839031409797829869_1258_r_000000_0' to file:/tmp/ReadsSparkSinkUnitTest1460954206127388021.bam.parts/_temporary/0/task_202510271738254839031409797829869_1258_r_000000
17:38:25.955 INFO SparkHadoopMapRedUtil - attempt_202510271738254839031409797829869_1258_r_000000_0: Committed. Elapsed time: 0 ms.
17:38:25.955 INFO Executor - Finished task 0.0 in stage 261.0 (TID 317). 1858 bytes result sent to driver
17:38:25.955 INFO TaskSetManager - Finished task 0.0 in stage 261.0 (TID 317) in 40 ms on localhost (executor driver) (1/1)
17:38:25.955 INFO TaskSchedulerImpl - Removed TaskSet 261.0, whose tasks have all completed, from pool
17:38:25.956 INFO DAGScheduler - ResultStage 261 (runJob at SparkHadoopWriter.scala:83) finished in 0.049 s
17:38:25.956 INFO DAGScheduler - Job 197 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:25.956 INFO TaskSchedulerImpl - Killing all running tasks in stage 261: Stage finished
17:38:25.956 INFO DAGScheduler - Job 197 finished: runJob at SparkHadoopWriter.scala:83, took 0.117808 s
17:38:25.956 INFO SparkHadoopWriter - Start to commit write Job job_202510271738254839031409797829869_1258.
17:38:25.960 INFO SparkHadoopWriter - Write Job job_202510271738254839031409797829869_1258 committed. Elapsed time: 3 ms.
17:38:25.962 WARN IntelDeflaterFactory - Intel Deflater not supported, using Java.util.zip.Deflater
17:38:25.967 INFO HadoopFileSystemWrapper - Concatenating 3 parts to file:////tmp/ReadsSparkSinkUnitTest1460954206127388021.bam
17:38:25.971 INFO HadoopFileSystemWrapper - Concatenating to file:////tmp/ReadsSparkSinkUnitTest1460954206127388021.bam done
17:38:25.971 INFO IndexFileMerger - Merging .sbi files in temp directory file:////tmp/ReadsSparkSinkUnitTest1460954206127388021.bam.parts/ to file:////tmp/ReadsSparkSinkUnitTest1460954206127388021.bam.sbi
17:38:25.975 INFO IndexFileMerger - Done merging .sbi files
17:38:25.975 INFO IndexFileMerger - Merging .bai files in temp directory file:////tmp/ReadsSparkSinkUnitTest1460954206127388021.bam.parts/ to file:////tmp/ReadsSparkSinkUnitTest1460954206127388021.bam.bai
17:38:25.979 INFO IndexFileMerger - Done merging .bai files
17:38:25.980 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:25.981 INFO MemoryStore - Block broadcast_531 stored as values in memory (estimated size 320.0 B, free 1918.4 MiB)
17:38:25.981 INFO MemoryStore - Block broadcast_531_piece0 stored as bytes in memory (estimated size 233.0 B, free 1918.4 MiB)
17:38:25.982 INFO BlockManagerInfo - Added broadcast_531_piece0 in memory on localhost:37953 (size: 233.0 B, free: 1919.7 MiB)
17:38:25.982 INFO SparkContext - Created broadcast 531 from broadcast at BamSource.java:104
17:38:25.983 INFO MemoryStore - Block broadcast_532 stored as values in memory (estimated size 297.9 KiB, free 1918.1 MiB)
17:38:25.988 INFO MemoryStore - Block broadcast_532_piece0 stored as bytes in memory (estimated size 50.2 KiB, free 1918.0 MiB)
17:38:25.989 INFO BlockManagerInfo - Added broadcast_532_piece0 in memory on localhost:37953 (size: 50.2 KiB, free: 1919.7 MiB)
17:38:25.989 INFO SparkContext - Created broadcast 532 from newAPIHadoopFile at PathSplitSource.java:96
17:38:25.998 INFO FileInputFormat - Total input files to process : 1
17:38:26.011 INFO SparkContext - Starting job: collect at ReadsSparkSinkUnitTest.java:182
17:38:26.011 INFO DAGScheduler - Got job 198 (collect at ReadsSparkSinkUnitTest.java:182) with 1 output partitions
17:38:26.011 INFO DAGScheduler - Final stage: ResultStage 262 (collect at ReadsSparkSinkUnitTest.java:182)
17:38:26.011 INFO DAGScheduler - Parents of final stage: List()
17:38:26.011 INFO DAGScheduler - Missing parents: List()
17:38:26.011 INFO DAGScheduler - Submitting ResultStage 262 (MapPartitionsRDD[1264] at filter at ReadsSparkSource.java:96), which has no missing parents
17:38:26.017 INFO MemoryStore - Block broadcast_533 stored as values in memory (estimated size 148.2 KiB, free 1917.9 MiB)
17:38:26.017 INFO MemoryStore - Block broadcast_533_piece0 stored as bytes in memory (estimated size 54.5 KiB, free 1917.8 MiB)
17:38:26.017 INFO BlockManagerInfo - Added broadcast_533_piece0 in memory on localhost:37953 (size: 54.5 KiB, free: 1919.6 MiB)
17:38:26.018 INFO SparkContext - Created broadcast 533 from broadcast at DAGScheduler.scala:1580
17:38:26.018 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 262 (MapPartitionsRDD[1264] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:38:26.018 INFO TaskSchedulerImpl - Adding task set 262.0 with 1 tasks resource profile 0
17:38:26.018 INFO TaskSetManager - Starting task 0.0 in stage 262.0 (TID 318) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7808 bytes)
17:38:26.019 INFO Executor - Running task 0.0 in stage 262.0 (TID 318)
17:38:26.031 INFO NewHadoopRDD - Input split: file:/tmp/ReadsSparkSinkUnitTest1460954206127388021.bam:0+238969
17:38:26.031 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:26.038 INFO BlockManagerInfo - Removed broadcast_529_piece0 on localhost:37953 in memory (size: 166.1 KiB, free: 1919.8 MiB)
17:38:26.039 INFO Executor - Finished task 0.0 in stage 262.0 (TID 318). 651569 bytes result sent to driver
17:38:26.039 INFO BlockManagerInfo - Removed broadcast_527_piece0 on localhost:37953 in memory (size: 9.6 KiB, free: 1919.8 MiB)
17:38:26.040 INFO BlockManagerInfo - Removed broadcast_528_piece0 on localhost:37953 in memory (size: 9.6 KiB, free: 1919.8 MiB)
17:38:26.040 INFO TaskSetManager - Finished task 0.0 in stage 262.0 (TID 318) in 22 ms on localhost (executor driver) (1/1)
17:38:26.040 INFO BlockManagerInfo - Removed broadcast_530_piece0 on localhost:37953 in memory (size: 66.9 KiB, free: 1919.8 MiB)
17:38:26.040 INFO TaskSchedulerImpl - Removed TaskSet 262.0, whose tasks have all completed, from pool
17:38:26.040 INFO DAGScheduler - ResultStage 262 (collect at ReadsSparkSinkUnitTest.java:182) finished in 0.028 s
17:38:26.040 INFO DAGScheduler - Job 198 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:26.040 INFO TaskSchedulerImpl - Killing all running tasks in stage 262: Stage finished
17:38:26.041 INFO DAGScheduler - Job 198 finished: collect at ReadsSparkSinkUnitTest.java:182, took 0.029709 s
17:38:26.051 INFO SparkContext - Starting job: count at ReadsSparkSinkUnitTest.java:185
17:38:26.051 INFO DAGScheduler - Got job 199 (count at ReadsSparkSinkUnitTest.java:185) with 1 output partitions
17:38:26.051 INFO DAGScheduler - Final stage: ResultStage 263 (count at ReadsSparkSinkUnitTest.java:185)
17:38:26.051 INFO DAGScheduler - Parents of final stage: List()
17:38:26.051 INFO DAGScheduler - Missing parents: List()
17:38:26.051 INFO DAGScheduler - Submitting ResultStage 263 (MapPartitionsRDD[1246] at filter at ReadsSparkSource.java:96), which has no missing parents
17:38:26.067 INFO MemoryStore - Block broadcast_534 stored as values in memory (estimated size 426.1 KiB, free 1918.7 MiB)
17:38:26.068 INFO MemoryStore - Block broadcast_534_piece0 stored as bytes in memory (estimated size 153.6 KiB, free 1918.6 MiB)
17:38:26.068 INFO BlockManagerInfo - Added broadcast_534_piece0 in memory on localhost:37953 (size: 153.6 KiB, free: 1919.7 MiB)
17:38:26.068 INFO SparkContext - Created broadcast 534 from broadcast at DAGScheduler.scala:1580
17:38:26.068 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 263 (MapPartitionsRDD[1246] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:38:26.068 INFO TaskSchedulerImpl - Adding task set 263.0 with 1 tasks resource profile 0
17:38:26.069 INFO TaskSetManager - Starting task 0.0 in stage 263.0 (TID 319) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7872 bytes)
17:38:26.069 INFO Executor - Running task 0.0 in stage 263.0 (TID 319)
17:38:26.102 INFO NewHadoopRDD - Input split: file:/home/runner/work/gatk/gatk/src/test/resources/org/broadinstitute/hellbender/tools/BQSR/HiSeq.1mb.1RG.2k_lines.bam:0+222075
17:38:26.103 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:26.104 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:26.111 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:26.115 INFO Executor - Finished task 0.0 in stage 263.0 (TID 319). 989 bytes result sent to driver
17:38:26.115 INFO TaskSetManager - Finished task 0.0 in stage 263.0 (TID 319) in 46 ms on localhost (executor driver) (1/1)
17:38:26.115 INFO TaskSchedulerImpl - Removed TaskSet 263.0, whose tasks have all completed, from pool
17:38:26.115 INFO DAGScheduler - ResultStage 263 (count at ReadsSparkSinkUnitTest.java:185) finished in 0.064 s
17:38:26.115 INFO DAGScheduler - Job 199 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:26.115 INFO TaskSchedulerImpl - Killing all running tasks in stage 263: Stage finished
17:38:26.115 INFO DAGScheduler - Job 199 finished: count at ReadsSparkSinkUnitTest.java:185, took 0.064633 s
17:38:26.118 INFO SparkContext - Starting job: count at ReadsSparkSinkUnitTest.java:185
17:38:26.119 INFO DAGScheduler - Got job 200 (count at ReadsSparkSinkUnitTest.java:185) with 1 output partitions
17:38:26.119 INFO DAGScheduler - Final stage: ResultStage 264 (count at ReadsSparkSinkUnitTest.java:185)
17:38:26.119 INFO DAGScheduler - Parents of final stage: List()
17:38:26.119 INFO DAGScheduler - Missing parents: List()
17:38:26.119 INFO DAGScheduler - Submitting ResultStage 264 (MapPartitionsRDD[1264] at filter at ReadsSparkSource.java:96), which has no missing parents
17:38:26.124 INFO MemoryStore - Block broadcast_535 stored as values in memory (estimated size 148.1 KiB, free 1918.4 MiB)
17:38:26.125 INFO MemoryStore - Block broadcast_535_piece0 stored as bytes in memory (estimated size 54.5 KiB, free 1918.4 MiB)
17:38:26.125 INFO BlockManagerInfo - Added broadcast_535_piece0 in memory on localhost:37953 (size: 54.5 KiB, free: 1919.6 MiB)
17:38:26.125 INFO SparkContext - Created broadcast 535 from broadcast at DAGScheduler.scala:1580
17:38:26.125 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 264 (MapPartitionsRDD[1264] at filter at ReadsSparkSource.java:96) (first 15 tasks are for partitions Vector(0))
17:38:26.125 INFO TaskSchedulerImpl - Adding task set 264.0 with 1 tasks resource profile 0
17:38:26.126 INFO TaskSetManager - Starting task 0.0 in stage 264.0 (TID 320) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7808 bytes)
17:38:26.126 INFO Executor - Running task 0.0 in stage 264.0 (TID 320)
17:38:26.138 INFO NewHadoopRDD - Input split: file:/tmp/ReadsSparkSinkUnitTest1460954206127388021.bam:0+238969
17:38:26.138 WARN IntelInflaterFactory - IntelInflater is not supported, using Java.util.zip.Inflater
17:38:26.142 INFO Executor - Finished task 0.0 in stage 264.0 (TID 320). 989 bytes result sent to driver
17:38:26.142 INFO TaskSetManager - Finished task 0.0 in stage 264.0 (TID 320) in 16 ms on localhost (executor driver) (1/1)
17:38:26.142 INFO TaskSchedulerImpl - Removed TaskSet 264.0, whose tasks have all completed, from pool
17:38:26.142 INFO DAGScheduler - ResultStage 264 (count at ReadsSparkSinkUnitTest.java:185) finished in 0.023 s
17:38:26.142 INFO DAGScheduler - Job 200 is finished. Cancelling potential speculative or zombie tasks for this job
17:38:26.142 INFO TaskSchedulerImpl - Killing all running tasks in stage 264: Stage finished
17:38:26.142 INFO DAGScheduler - Job 200 finished: count at ReadsSparkSinkUnitTest.java:185, took 0.023942 s