Class org.broadinstitute.hellbender.engine.spark.SparkSharderUnitTest

21:17:54.235 INFO  SparkContext - Starting job: collect at SparkSharder.java:388
21:17:54.251 INFO  DAGScheduler - Got job 0 (collect at SparkSharder.java:388) with 1 output partitions
21:17:54.251 INFO  DAGScheduler - Final stage: ResultStage 0 (collect at SparkSharder.java:388)
21:17:54.251 INFO  DAGScheduler - Parents of final stage: List()
21:17:54.253 INFO  DAGScheduler - Missing parents: List()
21:17:54.255 INFO  DAGScheduler - Submitting ResultStage 0 (MapPartitionsRDD[3] at mapPartitions at SparkSharder.java:386), which has no missing parents
21:17:54.293 INFO  MemoryStore - Block broadcast_0 stored as values in memory (estimated size 4.5 KiB, free 1920.0 MiB)
21:17:54.319 INFO  MemoryStore - Block broadcast_0_piece0 stored as bytes in memory (estimated size 2.4 KiB, free 1920.0 MiB)
21:17:54.321 INFO  BlockManagerInfo - Added broadcast_0_piece0 in memory on localhost:40577 (size: 2.4 KiB, free: 1920.0 MiB)
21:17:54.324 INFO  SparkContext - Created broadcast 0 from broadcast at DAGScheduler.scala:1580
21:17:54.336 INFO  DAGScheduler - Submitting 1 missing tasks from ResultStage 0 (MapPartitionsRDD[3] at mapPartitions at SparkSharder.java:386) (first 15 tasks are for partitions Vector(0))
21:17:54.336 INFO  TaskSchedulerImpl - Adding task set 0.0 with 1 tasks resource profile 0
21:17:54.382 INFO  TaskSetManager - Starting task 0.0 in stage 0.0 (TID 0) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7718 bytes) 
21:17:54.394 INFO  Executor - Running task 0.0 in stage 0.0 (TID 0)
21:17:54.462 INFO  Executor - Finished task 0.0 in stage 0.0 (TID 0). 1116 bytes result sent to driver
21:17:54.468 INFO  TaskSetManager - Finished task 0.0 in stage 0.0 (TID 0) in 106 ms on localhost (executor driver) (1/1)
21:17:54.469 INFO  TaskSchedulerImpl - Removed TaskSet 0.0, whose tasks have all completed, from pool 
21:17:54.472 INFO  DAGScheduler - ResultStage 0 (collect at SparkSharder.java:388) finished in 0.204 s
21:17:54.474 INFO  DAGScheduler - Job 0 is finished. Cancelling potential speculative or zombie tasks for this job
21:17:54.474 INFO  TaskSchedulerImpl - Killing all running tasks in stage 0: Stage finished
21:17:54.475 INFO  DAGScheduler - Job 0 finished: collect at SparkSharder.java:388, took 0.240704 s
21:17:54.479 INFO  MemoryStore - Block broadcast_1 stored as values in memory (estimated size 248.0 B, free 1920.0 MiB)
21:17:54.483 INFO  MemoryStore - Block broadcast_1_piece0 stored as bytes in memory (estimated size 143.0 B, free 1920.0 MiB)
21:17:54.483 INFO  BlockManagerInfo - Added broadcast_1_piece0 in memory on localhost:40577 (size: 143.0 B, free: 1920.0 MiB)
21:17:54.484 INFO  SparkContext - Created broadcast 1 from broadcast at SparkSharder.java:214
21:17:54.486 INFO  MemoryStore - Block broadcast_2 stored as values in memory (estimated size 608.0 B, free 1920.0 MiB)
21:17:54.488 INFO  MemoryStore - Block broadcast_2_piece0 stored as bytes in memory (estimated size 349.0 B, free 1920.0 MiB)
21:17:54.489 INFO  BlockManagerInfo - Added broadcast_2_piece0 in memory on localhost:40577 (size: 349.0 B, free: 1920.0 MiB)
21:17:54.489 INFO  SparkContext - Created broadcast 2 from broadcast at SparkSharder.java:221
21:17:54.531 INFO  SparkContext - Starting job: collectAsMap at SparkSharder.java:258
21:17:54.540 INFO  DAGScheduler - Registering RDD 5 (mapToPair at SparkSharder.java:247) as input to shuffle 1
21:17:54.544 INFO  DAGScheduler - Registering RDD 8 (mapToPair at SparkSharder.java:255) as input to shuffle 0
21:17:54.544 INFO  DAGScheduler - Got job 1 (collectAsMap at SparkSharder.java:258) with 1 output partitions
21:17:54.544 INFO  DAGScheduler - Final stage: ResultStage 3 (collectAsMap at SparkSharder.java:258)
21:17:54.544 INFO  DAGScheduler - Parents of final stage: List(ShuffleMapStage 2)
21:17:54.545 INFO  DAGScheduler - Missing parents: List(ShuffleMapStage 2)
21:17:54.548 INFO  DAGScheduler - Submitting ShuffleMapStage 1 (MapPartitionsRDD[5] at mapToPair at SparkSharder.java:247), which has no missing parents
21:17:54.559 INFO  MemoryStore - Block broadcast_3 stored as values in memory (estimated size 7.9 KiB, free 1920.0 MiB)
21:17:54.561 INFO  MemoryStore - Block broadcast_3_piece0 stored as bytes in memory (estimated size 4.3 KiB, free 1920.0 MiB)
21:17:54.562 INFO  BlockManagerInfo - Added broadcast_3_piece0 in memory on localhost:40577 (size: 4.3 KiB, free: 1920.0 MiB)
21:17:54.562 INFO  SparkContext - Created broadcast 3 from broadcast at DAGScheduler.scala:1580
21:17:54.564 INFO  DAGScheduler - Submitting 4 missing tasks from ShuffleMapStage 1 (MapPartitionsRDD[5] at mapToPair at SparkSharder.java:247) (first 15 tasks are for partitions Vector(0, 1, 2, 3))
21:17:54.564 INFO  TaskSchedulerImpl - Adding task set 1.0 with 4 tasks resource profile 0
21:17:54.566 INFO  TaskSetManager - Starting task 0.0 in stage 1.0 (TID 1) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7569 bytes) 
21:17:54.567 INFO  TaskSetManager - Starting task 1.0 in stage 1.0 (TID 2) (localhost, executor driver, partition 1, PROCESS_LOCAL, 7636 bytes) 
21:17:54.567 INFO  TaskSetManager - Starting task 2.0 in stage 1.0 (TID 3) (localhost, executor driver, partition 2, PROCESS_LOCAL, 7636 bytes) 
21:17:54.568 INFO  TaskSetManager - Starting task 3.0 in stage 1.0 (TID 4) (localhost, executor driver, partition 3, PROCESS_LOCAL, 7636 bytes) 
21:17:54.568 INFO  Executor - Running task 0.0 in stage 1.0 (TID 1)
21:17:54.568 INFO  Executor - Running task 1.0 in stage 1.0 (TID 2)
21:17:54.571 INFO  Executor - Running task 3.0 in stage 1.0 (TID 4)
21:17:54.571 INFO  Executor - Running task 2.0 in stage 1.0 (TID 3)
21:17:54.591 INFO  Executor - Finished task 0.0 in stage 1.0 (TID 1). 933 bytes result sent to driver
21:17:54.604 INFO  TaskSetManager - Finished task 0.0 in stage 1.0 (TID 1) in 39 ms on localhost (executor driver) (1/4)
21:17:54.608 INFO  Executor - Finished task 2.0 in stage 1.0 (TID 3). 1062 bytes result sent to driver
21:17:54.609 INFO  Executor - Finished task 1.0 in stage 1.0 (TID 2). 1062 bytes result sent to driver
21:17:54.610 INFO  TaskSetManager - Finished task 2.0 in stage 1.0 (TID 3) in 43 ms on localhost (executor driver) (2/4)
21:17:54.611 INFO  TaskSetManager - Finished task 1.0 in stage 1.0 (TID 2) in 44 ms on localhost (executor driver) (3/4)
21:17:54.613 INFO  Executor - Finished task 3.0 in stage 1.0 (TID 4). 1105 bytes result sent to driver
21:17:54.615 INFO  TaskSetManager - Finished task 3.0 in stage 1.0 (TID 4) in 48 ms on localhost (executor driver) (4/4)
21:17:54.615 INFO  TaskSchedulerImpl - Removed TaskSet 1.0, whose tasks have all completed, from pool 
21:17:54.615 INFO  DAGScheduler - ShuffleMapStage 1 (mapToPair at SparkSharder.java:247) finished in 0.065 s
21:17:54.615 INFO  DAGScheduler - looking for newly runnable stages
21:17:54.616 INFO  DAGScheduler - running: HashSet()
21:17:54.616 INFO  DAGScheduler - waiting: HashSet(ShuffleMapStage 2, ResultStage 3)
21:17:54.616 INFO  DAGScheduler - failed: HashSet()
21:17:54.618 INFO  DAGScheduler - Submitting ShuffleMapStage 2 (MapPartitionsRDD[8] at mapToPair at SparkSharder.java:255), which has no missing parents
21:17:54.625 INFO  MemoryStore - Block broadcast_4 stored as values in memory (estimated size 8.2 KiB, free 1920.0 MiB)
21:17:54.628 INFO  MemoryStore - Block broadcast_4_piece0 stored as bytes in memory (estimated size 4.5 KiB, free 1920.0 MiB)
21:17:54.629 INFO  BlockManagerInfo - Added broadcast_4_piece0 in memory on localhost:40577 (size: 4.5 KiB, free: 1920.0 MiB)
21:17:54.629 INFO  SparkContext - Created broadcast 4 from broadcast at DAGScheduler.scala:1580
21:17:54.630 INFO  DAGScheduler - Submitting 1 missing tasks from ShuffleMapStage 2 (MapPartitionsRDD[8] at mapToPair at SparkSharder.java:255) (first 15 tasks are for partitions Vector(0))
21:17:54.630 INFO  TaskSchedulerImpl - Adding task set 2.0 with 1 tasks resource profile 0
21:17:54.634 INFO  TaskSetManager - Starting task 0.0 in stage 2.0 (TID 5) (localhost, executor driver, partition 0, NODE_LOCAL, 7502 bytes) 
21:17:54.635 INFO  Executor - Running task 0.0 in stage 2.0 (TID 5)
21:17:54.672 INFO  ShuffleBlockFetcherIterator - Getting 3 (513.0 B) non-empty blocks including 3 (513.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
21:17:54.674 INFO  ShuffleBlockFetcherIterator - Started 0 remote fetches in 8 ms
21:17:54.706 INFO  MemoryStore - Block rdd_7_0 stored as values in memory (estimated size 392.0 B, free 1920.0 MiB)
21:17:54.706 INFO  BlockManagerInfo - Added rdd_7_0 in memory on localhost:40577 (size: 392.0 B, free: 1920.0 MiB)
21:17:54.719 INFO  Executor - Finished task 0.0 in stage 2.0 (TID 5). 1922 bytes result sent to driver
21:17:54.721 INFO  TaskSetManager - Finished task 0.0 in stage 2.0 (TID 5) in 88 ms on localhost (executor driver) (1/1)
21:17:54.721 INFO  TaskSchedulerImpl - Removed TaskSet 2.0, whose tasks have all completed, from pool 
21:17:54.722 INFO  DAGScheduler - ShuffleMapStage 2 (mapToPair at SparkSharder.java:255) finished in 0.099 s
21:17:54.722 INFO  DAGScheduler - looking for newly runnable stages
21:17:54.722 INFO  DAGScheduler - running: HashSet()
21:17:54.722 INFO  DAGScheduler - waiting: HashSet(ResultStage 3)
21:17:54.722 INFO  DAGScheduler - failed: HashSet()
21:17:54.723 INFO  DAGScheduler - Submitting ResultStage 3 (ShuffledRDD[9] at reduceByKey at SparkSharder.java:257), which has no missing parents
21:17:54.725 INFO  MemoryStore - Block broadcast_5 stored as values in memory (estimated size 5.8 KiB, free 1920.0 MiB)
21:17:54.728 INFO  MemoryStore - Block broadcast_5_piece0 stored as bytes in memory (estimated size 3.2 KiB, free 1920.0 MiB)
21:17:54.729 INFO  BlockManagerInfo - Added broadcast_5_piece0 in memory on localhost:40577 (size: 3.2 KiB, free: 1920.0 MiB)
21:17:54.729 INFO  SparkContext - Created broadcast 5 from broadcast at DAGScheduler.scala:1580
21:17:54.730 INFO  DAGScheduler - Submitting 1 missing tasks from ResultStage 3 (ShuffledRDD[9] at reduceByKey at SparkSharder.java:257) (first 15 tasks are for partitions Vector(0))
21:17:54.730 INFO  TaskSchedulerImpl - Adding task set 3.0 with 1 tasks resource profile 0
21:17:54.732 INFO  TaskSetManager - Starting task 0.0 in stage 3.0 (TID 6) (localhost, executor driver, partition 0, NODE_LOCAL, 7513 bytes) 
21:17:54.733 INFO  Executor - Running task 0.0 in stage 3.0 (TID 6)
21:17:54.737 INFO  ShuffleBlockFetcherIterator - Getting 1 (54.0 B) non-empty blocks including 1 (54.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
21:17:54.737 INFO  ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
21:17:54.749 INFO  Executor - Finished task 0.0 in stage 3.0 (TID 6). 1770 bytes result sent to driver
21:17:54.750 INFO  TaskSetManager - Finished task 0.0 in stage 3.0 (TID 6) in 19 ms on localhost (executor driver) (1/1)
21:17:54.750 INFO  TaskSchedulerImpl - Removed TaskSet 3.0, whose tasks have all completed, from pool 
21:17:54.751 INFO  DAGScheduler - ResultStage 3 (collectAsMap at SparkSharder.java:258) finished in 0.028 s
21:17:54.751 INFO  DAGScheduler - Job 1 is finished. Cancelling potential speculative or zombie tasks for this job
21:17:54.751 INFO  TaskSchedulerImpl - Killing all running tasks in stage 3: Stage finished
21:17:54.752 INFO  DAGScheduler - Job 1 finished: collectAsMap at SparkSharder.java:258, took 0.220946 s
21:17:54.789 INFO  SparkContext - Starting job: collectAsMap at SparkSharderUnitTest.java:305
21:17:54.791 INFO  DAGScheduler - Got job 2 (collectAsMap at SparkSharderUnitTest.java:305) with 1 output partitions
21:17:54.791 INFO  DAGScheduler - Final stage: ResultStage 5 (collectAsMap at SparkSharderUnitTest.java:305)
21:17:54.791 INFO  DAGScheduler - Parents of final stage: List(ShuffleMapStage 4)
21:17:54.792 INFO  DAGScheduler - Missing parents: List()
21:17:54.792 INFO  DAGScheduler - Submitting ResultStage 5 (MapPartitionsRDD[13] at flatMapToPair at SparkSharderUnitTest.java:304), which has no missing parents
21:17:54.796 INFO  MemoryStore - Block broadcast_6 stored as values in memory (estimated size 10.2 KiB, free 1919.9 MiB)
21:17:54.801 INFO  MemoryStore - Block broadcast_6_piece0 stored as bytes in memory (estimated size 5.0 KiB, free 1919.9 MiB)
21:17:54.803 INFO  BlockManagerInfo - Added broadcast_6_piece0 in memory on localhost:40577 (size: 5.0 KiB, free: 1920.0 MiB)
21:17:54.803 INFO  SparkContext - Created broadcast 6 from broadcast at DAGScheduler.scala:1580
21:17:54.804 INFO  DAGScheduler - Submitting 1 missing tasks from ResultStage 5 (MapPartitionsRDD[13] at flatMapToPair at SparkSharderUnitTest.java:304) (first 15 tasks are for partitions Vector(0))
21:17:54.805 INFO  TaskSchedulerImpl - Adding task set 5.0 with 1 tasks resource profile 0
21:17:54.811 INFO  TaskSetManager - Starting task 0.0 in stage 5.0 (TID 7) (localhost, executor driver, partition 0, PROCESS_LOCAL, 8559 bytes) 
21:17:54.812 INFO  Executor - Running task 0.0 in stage 5.0 (TID 7)
21:17:54.827 INFO  BlockManager - Found block rdd_7_0 locally
21:17:54.830 INFO  Executor - Finished task 0.0 in stage 5.0 (TID 7). 1097 bytes result sent to driver
21:17:54.832 INFO  TaskSetManager - Finished task 0.0 in stage 5.0 (TID 7) in 25 ms on localhost (executor driver) (1/1)
21:17:54.832 INFO  TaskSchedulerImpl - Removed TaskSet 5.0, whose tasks have all completed, from pool 
21:17:54.833 INFO  DAGScheduler - ResultStage 5 (collectAsMap at SparkSharderUnitTest.java:305) finished in 0.040 s
21:17:54.833 INFO  DAGScheduler - Job 2 is finished. Cancelling potential speculative or zombie tasks for this job
21:17:54.833 INFO  TaskSchedulerImpl - Killing all running tasks in stage 5: Stage finished
21:17:54.833 INFO  DAGScheduler - Job 2 finished: collectAsMap at SparkSharderUnitTest.java:305, took 0.043776 s
21:17:54.835 INFO  MemoryStore - Block broadcast_7 stored as values in memory (estimated size 720.0 B, free 1919.9 MiB)
21:17:54.840 INFO  MemoryStore - Block broadcast_7_piece0 stored as bytes in memory (estimated size 326.0 B, free 1919.9 MiB)
21:17:54.842 INFO  BlockManagerInfo - Added broadcast_7_piece0 in memory on localhost:40577 (size: 326.0 B, free: 1920.0 MiB)
21:17:54.842 INFO  SparkContext - Created broadcast 7 from broadcast at SparkSharder.java:100
21:17:54.879 INFO  SparkContext - Starting job: collectAsMap at SparkSharderUnitTest.java:310
21:17:54.880 INFO  DAGScheduler - Registering RDD 14 (flatMapToPair at SparkSharder.java:101) as input to shuffle 2
21:17:54.880 INFO  DAGScheduler - Got job 3 (collectAsMap at SparkSharderUnitTest.java:310) with 1 output partitions
21:17:54.880 INFO  DAGScheduler - Final stage: ResultStage 7 (collectAsMap at SparkSharderUnitTest.java:310)
21:17:54.880 INFO  DAGScheduler - Parents of final stage: List(ShuffleMapStage 6)
21:17:54.880 INFO  DAGScheduler - Missing parents: List(ShuffleMapStage 6)
21:17:54.882 INFO  DAGScheduler - Submitting ShuffleMapStage 6 (MapPartitionsRDD[14] at flatMapToPair at SparkSharder.java:101), which has no missing parents
21:17:54.884 INFO  MemoryStore - Block broadcast_8 stored as values in memory (estimated size 7.2 KiB, free 1919.9 MiB)
21:17:54.888 INFO  MemoryStore - Block broadcast_8_piece0 stored as bytes in memory (estimated size 3.8 KiB, free 1919.9 MiB)
21:17:54.889 INFO  BlockManagerInfo - Added broadcast_8_piece0 in memory on localhost:40577 (size: 3.8 KiB, free: 1920.0 MiB)
21:17:54.889 INFO  SparkContext - Created broadcast 8 from broadcast at DAGScheduler.scala:1580
21:17:54.891 INFO  DAGScheduler - Submitting 1 missing tasks from ShuffleMapStage 6 (MapPartitionsRDD[14] at flatMapToPair at SparkSharder.java:101) (first 15 tasks are for partitions Vector(0))
21:17:54.891 INFO  TaskSchedulerImpl - Adding task set 6.0 with 1 tasks resource profile 0
21:17:54.893 INFO  TaskSetManager - Starting task 0.0 in stage 6.0 (TID 8) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7707 bytes) 
21:17:54.895 INFO  Executor - Running task 0.0 in stage 6.0 (TID 8)
21:17:54.920 INFO  Executor - Finished task 0.0 in stage 6.0 (TID 8). 1062 bytes result sent to driver
21:17:54.920 INFO  TaskSetManager - Finished task 0.0 in stage 6.0 (TID 8) in 28 ms on localhost (executor driver) (1/1)
21:17:54.921 INFO  TaskSchedulerImpl - Removed TaskSet 6.0, whose tasks have all completed, from pool 
21:17:54.922 INFO  DAGScheduler - ShuffleMapStage 6 (flatMapToPair at SparkSharder.java:101) finished in 0.039 s
21:17:54.923 INFO  DAGScheduler - looking for newly runnable stages
21:17:54.923 INFO  DAGScheduler - running: HashSet()
21:17:54.923 INFO  DAGScheduler - waiting: HashSet(ResultStage 7)
21:17:54.923 INFO  DAGScheduler - failed: HashSet()
21:17:54.923 INFO  DAGScheduler - Submitting ResultStage 7 (MapPartitionsRDD[18] at flatMapToPair at SparkSharderUnitTest.java:309), which has no missing parents
21:17:54.927 INFO  MemoryStore - Block broadcast_9 stored as values in memory (estimated size 9.4 KiB, free 1919.9 MiB)
21:17:54.930 INFO  MemoryStore - Block broadcast_9_piece0 stored as bytes in memory (estimated size 4.7 KiB, free 1919.9 MiB)
21:17:54.931 INFO  BlockManagerInfo - Added broadcast_9_piece0 in memory on localhost:40577 (size: 4.7 KiB, free: 1920.0 MiB)
21:17:54.932 INFO  SparkContext - Created broadcast 9 from broadcast at DAGScheduler.scala:1580
21:17:54.932 INFO  DAGScheduler - Submitting 1 missing tasks from ResultStage 7 (MapPartitionsRDD[18] at flatMapToPair at SparkSharderUnitTest.java:309) (first 15 tasks are for partitions Vector(0))
21:17:54.932 INFO  TaskSchedulerImpl - Adding task set 7.0 with 1 tasks resource profile 0
21:17:54.934 INFO  TaskSetManager - Starting task 0.0 in stage 7.0 (TID 9) (localhost, executor driver, partition 0, NODE_LOCAL, 7513 bytes) 
21:17:54.935 INFO  Executor - Running task 0.0 in stage 7.0 (TID 9)
21:17:54.940 INFO  ShuffleBlockFetcherIterator - Getting 1 (228.0 B) non-empty blocks including 1 (228.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
21:17:54.940 INFO  ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
21:17:54.954 INFO  Executor - Finished task 0.0 in stage 7.0 (TID 9). 1871 bytes result sent to driver
21:17:54.955 INFO  TaskSetManager - Finished task 0.0 in stage 7.0 (TID 9) in 21 ms on localhost (executor driver) (1/1)
21:17:54.955 INFO  TaskSchedulerImpl - Removed TaskSet 7.0, whose tasks have all completed, from pool 
21:17:54.955 INFO  DAGScheduler - ResultStage 7 (collectAsMap at SparkSharderUnitTest.java:310) finished in 0.031 s
21:17:54.956 INFO  DAGScheduler - Job 3 is finished. Cancelling potential speculative or zombie tasks for this job
21:17:54.956 INFO  TaskSchedulerImpl - Killing all running tasks in stage 7: Stage finished
21:17:54.956 INFO  DAGScheduler - Job 3 finished: collectAsMap at SparkSharderUnitTest.java:310, took 0.076660 s
21:17:54.973 INFO  SparkContext - Starting job: collect at SparkSharder.java:388
21:17:54.974 INFO  DAGScheduler - Got job 4 (collect at SparkSharder.java:388) with 1 output partitions
21:17:54.974 INFO  DAGScheduler - Final stage: ResultStage 8 (collect at SparkSharder.java:388)
21:17:54.974 INFO  DAGScheduler - Parents of final stage: List()
21:17:54.974 INFO  DAGScheduler - Missing parents: List()
21:17:54.974 INFO  DAGScheduler - Submitting ResultStage 8 (MapPartitionsRDD[20] at mapPartitions at SparkSharder.java:386), which has no missing parents
21:17:54.976 INFO  MemoryStore - Block broadcast_10 stored as values in memory (estimated size 4.5 KiB, free 1919.9 MiB)
21:17:54.979 INFO  MemoryStore - Block broadcast_10_piece0 stored as bytes in memory (estimated size 2.4 KiB, free 1919.9 MiB)
21:17:54.980 INFO  BlockManagerInfo - Added broadcast_10_piece0 in memory on localhost:40577 (size: 2.4 KiB, free: 1920.0 MiB)
21:17:54.980 INFO  SparkContext - Created broadcast 10 from broadcast at DAGScheduler.scala:1580
21:17:54.981 INFO  DAGScheduler - Submitting 1 missing tasks from ResultStage 8 (MapPartitionsRDD[20] at mapPartitions at SparkSharder.java:386) (first 15 tasks are for partitions Vector(0))
21:17:54.981 INFO  TaskSchedulerImpl - Adding task set 8.0 with 1 tasks resource profile 0
21:17:54.983 INFO  TaskSetManager - Starting task 0.0 in stage 8.0 (TID 10) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7730 bytes) 
21:17:54.983 INFO  Executor - Running task 0.0 in stage 8.0 (TID 10)
21:17:54.989 INFO  Executor - Finished task 0.0 in stage 8.0 (TID 10). 1073 bytes result sent to driver
21:17:54.992 INFO  TaskSetManager - Finished task 0.0 in stage 8.0 (TID 10) in 9 ms on localhost (executor driver) (1/1)
21:17:54.992 INFO  TaskSchedulerImpl - Removed TaskSet 8.0, whose tasks have all completed, from pool 
21:17:54.992 INFO  DAGScheduler - ResultStage 8 (collect at SparkSharder.java:388) finished in 0.017 s
21:17:54.992 INFO  DAGScheduler - Job 4 is finished. Cancelling potential speculative or zombie tasks for this job
21:17:54.992 INFO  TaskSchedulerImpl - Killing all running tasks in stage 8: Stage finished
21:17:54.992 INFO  DAGScheduler - Job 4 finished: collect at SparkSharder.java:388, took 0.019075 s
21:17:55.003 INFO  SparkContext - Starting job: collect at SparkSharder.java:388
21:17:55.003 INFO  DAGScheduler - Got job 5 (collect at SparkSharder.java:388) with 2 output partitions
21:17:55.003 INFO  DAGScheduler - Final stage: ResultStage 9 (collect at SparkSharder.java:388)
21:17:55.003 INFO  DAGScheduler - Parents of final stage: List()
21:17:55.004 INFO  DAGScheduler - Missing parents: List()
21:17:55.004 INFO  DAGScheduler - Submitting ResultStage 9 (MapPartitionsRDD[22] at mapPartitions at SparkSharder.java:386), which has no missing parents
21:17:55.006 INFO  MemoryStore - Block broadcast_11 stored as values in memory (estimated size 4.5 KiB, free 1919.9 MiB)
21:17:55.009 INFO  MemoryStore - Block broadcast_11_piece0 stored as bytes in memory (estimated size 2.4 KiB, free 1919.9 MiB)
21:17:55.010 INFO  BlockManagerInfo - Added broadcast_11_piece0 in memory on localhost:40577 (size: 2.4 KiB, free: 1920.0 MiB)
21:17:55.010 INFO  SparkContext - Created broadcast 11 from broadcast at DAGScheduler.scala:1580
21:17:55.011 INFO  DAGScheduler - Submitting 2 missing tasks from ResultStage 9 (MapPartitionsRDD[22] at mapPartitions at SparkSharder.java:386) (first 15 tasks are for partitions Vector(0, 1))
21:17:55.011 INFO  TaskSchedulerImpl - Adding task set 9.0 with 2 tasks resource profile 0
21:17:55.012 INFO  TaskSetManager - Starting task 0.0 in stage 9.0 (TID 11) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7692 bytes) 
21:17:55.013 INFO  TaskSetManager - Starting task 1.0 in stage 9.0 (TID 12) (localhost, executor driver, partition 1, PROCESS_LOCAL, 7692 bytes) 
21:17:55.013 INFO  Executor - Running task 0.0 in stage 9.0 (TID 11)
21:17:55.015 INFO  Executor - Running task 1.0 in stage 9.0 (TID 12)
21:17:55.018 INFO  Executor - Finished task 0.0 in stage 9.0 (TID 11). 1073 bytes result sent to driver
21:17:55.019 INFO  TaskSetManager - Finished task 0.0 in stage 9.0 (TID 11) in 7 ms on localhost (executor driver) (1/2)
21:17:55.019 INFO  Executor - Finished task 1.0 in stage 9.0 (TID 12). 1073 bytes result sent to driver
21:17:55.020 INFO  TaskSetManager - Finished task 1.0 in stage 9.0 (TID 12) in 8 ms on localhost (executor driver) (2/2)
21:17:55.020 INFO  TaskSchedulerImpl - Removed TaskSet 9.0, whose tasks have all completed, from pool 
21:17:55.020 INFO  DAGScheduler - ResultStage 9 (collect at SparkSharder.java:388) finished in 0.015 s
21:17:55.021 INFO  DAGScheduler - Job 5 is finished. Cancelling potential speculative or zombie tasks for this job
21:17:55.021 INFO  TaskSchedulerImpl - Killing all running tasks in stage 9: Stage finished
21:17:55.021 INFO  DAGScheduler - Job 5 finished: collect at SparkSharder.java:388, took 0.017844 s
21:17:55.037 INFO  SparkContext - Starting job: collect at SparkSharder.java:388
21:17:55.038 INFO  DAGScheduler - Got job 6 (collect at SparkSharder.java:388) with 3 output partitions
21:17:55.038 INFO  DAGScheduler - Final stage: ResultStage 10 (collect at SparkSharder.java:388)
21:17:55.038 INFO  DAGScheduler - Parents of final stage: List()
21:17:55.038 INFO  DAGScheduler - Missing parents: List()
21:17:55.038 INFO  DAGScheduler - Submitting ResultStage 10 (MapPartitionsRDD[24] at mapPartitions at SparkSharder.java:386), which has no missing parents
21:17:55.042 INFO  MemoryStore - Block broadcast_12 stored as values in memory (estimated size 4.5 KiB, free 1919.9 MiB)
21:17:55.045 INFO  MemoryStore - Block broadcast_12_piece0 stored as bytes in memory (estimated size 2.4 KiB, free 1919.9 MiB)
21:17:55.046 INFO  BlockManagerInfo - Added broadcast_12_piece0 in memory on localhost:40577 (size: 2.4 KiB, free: 1920.0 MiB)
21:17:55.046 INFO  SparkContext - Created broadcast 12 from broadcast at DAGScheduler.scala:1580
21:17:55.047 INFO  DAGScheduler - Submitting 3 missing tasks from ResultStage 10 (MapPartitionsRDD[24] at mapPartitions at SparkSharder.java:386) (first 15 tasks are for partitions Vector(0, 1, 2))
21:17:55.047 INFO  TaskSchedulerImpl - Adding task set 10.0 with 3 tasks resource profile 0
21:17:55.049 INFO  TaskSetManager - Starting task 0.0 in stage 10.0 (TID 13) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7680 bytes) 
21:17:55.049 INFO  TaskSetManager - Starting task 1.0 in stage 10.0 (TID 14) (localhost, executor driver, partition 1, PROCESS_LOCAL, 7682 bytes) 
21:17:55.049 INFO  TaskSetManager - Starting task 2.0 in stage 10.0 (TID 15) (localhost, executor driver, partition 2, PROCESS_LOCAL, 7680 bytes) 
21:17:55.050 INFO  Executor - Running task 1.0 in stage 10.0 (TID 14)
21:17:55.050 INFO  Executor - Running task 0.0 in stage 10.0 (TID 13)
21:17:55.052 INFO  Executor - Running task 2.0 in stage 10.0 (TID 15)
21:17:55.060 INFO  Executor - Finished task 1.0 in stage 10.0 (TID 14). 1116 bytes result sent to driver
21:17:55.062 INFO  Executor - Finished task 2.0 in stage 10.0 (TID 15). 1073 bytes result sent to driver
21:17:55.065 INFO  Executor - Finished task 0.0 in stage 10.0 (TID 13). 1073 bytes result sent to driver
21:17:55.065 INFO  TaskSetManager - Finished task 1.0 in stage 10.0 (TID 14) in 16 ms on localhost (executor driver) (1/3)
21:17:55.065 INFO  TaskSetManager - Finished task 2.0 in stage 10.0 (TID 15) in 16 ms on localhost (executor driver) (2/3)
21:17:55.066 INFO  TaskSetManager - Finished task 0.0 in stage 10.0 (TID 13) in 18 ms on localhost (executor driver) (3/3)
21:17:55.066 INFO  TaskSchedulerImpl - Removed TaskSet 10.0, whose tasks have all completed, from pool 
21:17:55.067 INFO  DAGScheduler - ResultStage 10 (collect at SparkSharder.java:388) finished in 0.028 s
21:17:55.067 INFO  DAGScheduler - Job 6 is finished. Cancelling potential speculative or zombie tasks for this job
21:17:55.067 INFO  TaskSchedulerImpl - Killing all running tasks in stage 10: Stage finished
21:17:55.067 INFO  DAGScheduler - Job 6 finished: collect at SparkSharder.java:388, took 0.030440 s
21:17:55.088 INFO  SparkContext - Starting job: collect at SparkSharder.java:388
21:17:55.088 INFO  DAGScheduler - Got job 7 (collect at SparkSharder.java:388) with 1 output partitions
21:17:55.088 INFO  DAGScheduler - Final stage: ResultStage 11 (collect at SparkSharder.java:388)
21:17:55.088 INFO  DAGScheduler - Parents of final stage: List()
21:17:55.088 INFO  DAGScheduler - Missing parents: List()
21:17:55.089 INFO  DAGScheduler - Submitting ResultStage 11 (MapPartitionsRDD[26] at mapPartitions at SparkSharder.java:386), which has no missing parents
21:17:55.092 INFO  MemoryStore - Block broadcast_13 stored as values in memory (estimated size 4.5 KiB, free 1919.9 MiB)
21:17:55.098 INFO  MemoryStore - Block broadcast_13_piece0 stored as bytes in memory (estimated size 2.4 KiB, free 1919.9 MiB)
21:17:55.098 INFO  BlockManagerInfo - Added broadcast_13_piece0 in memory on localhost:40577 (size: 2.4 KiB, free: 1920.0 MiB)
21:17:55.099 INFO  SparkContext - Created broadcast 13 from broadcast at DAGScheduler.scala:1580
21:17:55.099 INFO  DAGScheduler - Submitting 1 missing tasks from ResultStage 11 (MapPartitionsRDD[26] at mapPartitions at SparkSharder.java:386) (first 15 tasks are for partitions Vector(0))
21:17:55.099 INFO  TaskSchedulerImpl - Adding task set 11.0 with 1 tasks resource profile 0
21:17:55.101 INFO  TaskSetManager - Starting task 0.0 in stage 11.0 (TID 16) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7730 bytes) 
21:17:55.101 INFO  Executor - Running task 0.0 in stage 11.0 (TID 16)
21:17:55.106 INFO  Executor - Finished task 0.0 in stage 11.0 (TID 16). 1073 bytes result sent to driver
21:17:55.107 INFO  TaskSetManager - Finished task 0.0 in stage 11.0 (TID 16) in 7 ms on localhost (executor driver) (1/1)
21:17:55.108 INFO  DAGScheduler - ResultStage 11 (collect at SparkSharder.java:388) finished in 0.018 s
21:17:55.108 INFO  DAGScheduler - Job 7 is finished. Cancelling potential speculative or zombie tasks for this job
21:17:55.108 INFO  TaskSchedulerImpl - Removed TaskSet 11.0, whose tasks have all completed, from pool 
21:17:55.109 INFO  TaskSchedulerImpl - Killing all running tasks in stage 11: Stage finished
21:17:55.109 INFO  DAGScheduler - Job 7 finished: collect at SparkSharder.java:388, took 0.020986 s
21:17:55.123 INFO  SparkContext - Starting job: collect at SparkSharder.java:388
21:17:55.124 INFO  DAGScheduler - Got job 8 (collect at SparkSharder.java:388) with 2 output partitions
21:17:55.124 INFO  DAGScheduler - Final stage: ResultStage 12 (collect at SparkSharder.java:388)
21:17:55.124 INFO  DAGScheduler - Parents of final stage: List()
21:17:55.124 INFO  DAGScheduler - Missing parents: List()
21:17:55.125 INFO  DAGScheduler - Submitting ResultStage 12 (MapPartitionsRDD[28] at mapPartitions at SparkSharder.java:386), which has no missing parents
21:17:55.126 INFO  MemoryStore - Block broadcast_14 stored as values in memory (estimated size 4.5 KiB, free 1919.9 MiB)
21:17:55.130 INFO  MemoryStore - Block broadcast_14_piece0 stored as bytes in memory (estimated size 2.4 KiB, free 1919.9 MiB)
21:17:55.130 INFO  BlockManagerInfo - Added broadcast_14_piece0 in memory on localhost:40577 (size: 2.4 KiB, free: 1920.0 MiB)
21:17:55.130 INFO  SparkContext - Created broadcast 14 from broadcast at DAGScheduler.scala:1580
21:17:55.131 INFO  DAGScheduler - Submitting 2 missing tasks from ResultStage 12 (MapPartitionsRDD[28] at mapPartitions at SparkSharder.java:386) (first 15 tasks are for partitions Vector(0, 1))
21:17:55.131 INFO  TaskSchedulerImpl - Adding task set 12.0 with 2 tasks resource profile 0
21:17:55.132 INFO  TaskSetManager - Starting task 0.0 in stage 12.0 (TID 17) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7692 bytes) 
21:17:55.133 INFO  TaskSetManager - Starting task 1.0 in stage 12.0 (TID 18) (localhost, executor driver, partition 1, PROCESS_LOCAL, 7692 bytes) 
21:17:55.134 INFO  Executor - Running task 0.0 in stage 12.0 (TID 17)
21:17:55.135 INFO  Executor - Running task 1.0 in stage 12.0 (TID 18)
21:17:55.141 INFO  Executor - Finished task 1.0 in stage 12.0 (TID 18). 1073 bytes result sent to driver
21:17:55.141 INFO  Executor - Finished task 0.0 in stage 12.0 (TID 17). 1073 bytes result sent to driver
21:17:55.142 INFO  TaskSetManager - Finished task 1.0 in stage 12.0 (TID 18) in 10 ms on localhost (executor driver) (1/2)
21:17:55.142 INFO  TaskSetManager - Finished task 0.0 in stage 12.0 (TID 17) in 10 ms on localhost (executor driver) (2/2)
21:17:55.143 INFO  TaskSchedulerImpl - Removed TaskSet 12.0, whose tasks have all completed, from pool 
21:17:55.145 INFO  DAGScheduler - ResultStage 12 (collect at SparkSharder.java:388) finished in 0.019 s
21:17:55.146 INFO  DAGScheduler - Job 8 is finished. Cancelling potential speculative or zombie tasks for this job
21:17:55.146 INFO  TaskSchedulerImpl - Killing all running tasks in stage 12: Stage finished
21:17:55.147 INFO  DAGScheduler - Job 8 finished: collect at SparkSharder.java:388, took 0.023696 s
21:17:55.163 INFO  SparkContext - Starting job: collect at SparkSharder.java:388
21:17:55.164 INFO  DAGScheduler - Got job 9 (collect at SparkSharder.java:388) with 1 output partitions
21:17:55.165 INFO  DAGScheduler - Final stage: ResultStage 13 (collect at SparkSharder.java:388)
21:17:55.165 INFO  DAGScheduler - Parents of final stage: List()
21:17:55.165 INFO  DAGScheduler - Missing parents: List()
21:17:55.168 INFO  DAGScheduler - Submitting ResultStage 13 (MapPartitionsRDD[30] at mapPartitions at SparkSharder.java:386), which has no missing parents
21:17:55.170 INFO  MemoryStore - Block broadcast_15 stored as values in memory (estimated size 4.5 KiB, free 1919.9 MiB)
21:17:55.171 INFO  MemoryStore - Block broadcast_15_piece0 stored as bytes in memory (estimated size 2.4 KiB, free 1919.9 MiB)
21:17:55.171 INFO  BlockManagerInfo - Added broadcast_15_piece0 in memory on localhost:40577 (size: 2.4 KiB, free: 1920.0 MiB)
21:17:55.172 INFO  SparkContext - Created broadcast 15 from broadcast at DAGScheduler.scala:1580
21:17:55.172 INFO  DAGScheduler - Submitting 1 missing tasks from ResultStage 13 (MapPartitionsRDD[30] at mapPartitions at SparkSharder.java:386) (first 15 tasks are for partitions Vector(0))
21:17:55.173 INFO  TaskSchedulerImpl - Adding task set 13.0 with 1 tasks resource profile 0
21:17:55.174 INFO  TaskSetManager - Starting task 0.0 in stage 13.0 (TID 19) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7730 bytes) 
21:17:55.175 INFO  Executor - Running task 0.0 in stage 13.0 (TID 19)
21:17:55.181 INFO  Executor - Finished task 0.0 in stage 13.0 (TID 19). 1073 bytes result sent to driver
21:17:55.182 INFO  TaskSetManager - Finished task 0.0 in stage 13.0 (TID 19) in 8 ms on localhost (executor driver) (1/1)
21:17:55.183 INFO  TaskSchedulerImpl - Removed TaskSet 13.0, whose tasks have all completed, from pool 
21:17:55.184 INFO  DAGScheduler - ResultStage 13 (collect at SparkSharder.java:388) finished in 0.016 s
21:17:55.185 INFO  DAGScheduler - Job 9 is finished. Cancelling potential speculative or zombie tasks for this job
21:17:55.185 INFO  TaskSchedulerImpl - Killing all running tasks in stage 13: Stage finished
21:17:55.185 INFO  DAGScheduler - Job 9 finished: collect at SparkSharder.java:388, took 0.021778 s
21:17:55.203 INFO  SparkContext - Starting job: collect at SparkSharder.java:388
21:17:55.204 INFO  DAGScheduler - Got job 10 (collect at SparkSharder.java:388) with 2 output partitions
21:17:55.204 INFO  DAGScheduler - Final stage: ResultStage 14 (collect at SparkSharder.java:388)
21:17:55.204 INFO  DAGScheduler - Parents of final stage: List()
21:17:55.204 INFO  DAGScheduler - Missing parents: List()
21:17:55.205 INFO  DAGScheduler - Submitting ResultStage 14 (MapPartitionsRDD[32] at mapPartitions at SparkSharder.java:386), which has no missing parents
21:17:55.208 INFO  MemoryStore - Block broadcast_16 stored as values in memory (estimated size 4.5 KiB, free 1919.9 MiB)
21:17:55.209 INFO  MemoryStore - Block broadcast_16_piece0 stored as bytes in memory (estimated size 2.4 KiB, free 1919.9 MiB)
21:17:55.210 INFO  BlockManagerInfo - Added broadcast_16_piece0 in memory on localhost:40577 (size: 2.4 KiB, free: 1920.0 MiB)
21:17:55.211 INFO  SparkContext - Created broadcast 16 from broadcast at DAGScheduler.scala:1580
21:17:55.211 INFO  DAGScheduler - Submitting 2 missing tasks from ResultStage 14 (MapPartitionsRDD[32] at mapPartitions at SparkSharder.java:386) (first 15 tasks are for partitions Vector(0, 1))
21:17:55.211 INFO  TaskSchedulerImpl - Adding task set 14.0 with 2 tasks resource profile 0
21:17:55.215 INFO  TaskSetManager - Starting task 0.0 in stage 14.0 (TID 20) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7692 bytes) 
21:17:55.216 INFO  TaskSetManager - Starting task 1.0 in stage 14.0 (TID 21) (localhost, executor driver, partition 1, PROCESS_LOCAL, 7692 bytes) 
21:17:55.218 INFO  Executor - Running task 0.0 in stage 14.0 (TID 20)
21:17:55.219 INFO  Executor - Running task 1.0 in stage 14.0 (TID 21)
21:17:55.221 INFO  Executor - Finished task 1.0 in stage 14.0 (TID 21). 1030 bytes result sent to driver
21:17:55.224 INFO  TaskSetManager - Finished task 1.0 in stage 14.0 (TID 21) in 9 ms on localhost (executor driver) (1/2)
21:17:55.225 INFO  Executor - Finished task 0.0 in stage 14.0 (TID 20). 1073 bytes result sent to driver
21:17:55.225 INFO  TaskSetManager - Finished task 0.0 in stage 14.0 (TID 20) in 10 ms on localhost (executor driver) (2/2)
21:17:55.225 INFO  TaskSchedulerImpl - Removed TaskSet 14.0, whose tasks have all completed, from pool 
21:17:55.226 INFO  DAGScheduler - ResultStage 14 (collect at SparkSharder.java:388) finished in 0.020 s
21:17:55.226 INFO  DAGScheduler - Job 10 is finished. Cancelling potential speculative or zombie tasks for this job
21:17:55.226 INFO  TaskSchedulerImpl - Killing all running tasks in stage 14: Stage finished
21:17:55.226 INFO  DAGScheduler - Job 10 finished: collect at SparkSharder.java:388, took 0.022897 s
21:17:55.242 INFO  SparkContext - Starting job: collect at SparkSharder.java:388
21:17:55.243 INFO  DAGScheduler - Got job 11 (collect at SparkSharder.java:388) with 3 output partitions
21:17:55.243 INFO  DAGScheduler - Final stage: ResultStage 15 (collect at SparkSharder.java:388)
21:17:55.243 INFO  DAGScheduler - Parents of final stage: List()
21:17:55.243 INFO  DAGScheduler - Missing parents: List()
21:17:55.243 INFO  DAGScheduler - Submitting ResultStage 15 (MapPartitionsRDD[34] at mapPartitions at SparkSharder.java:386), which has no missing parents
21:17:55.245 INFO  MemoryStore - Block broadcast_17 stored as values in memory (estimated size 4.5 KiB, free 1919.9 MiB)
21:17:55.245 INFO  MemoryStore - Block broadcast_17_piece0 stored as bytes in memory (estimated size 2.4 KiB, free 1919.9 MiB)
21:17:55.250 INFO  BlockManagerInfo - Added broadcast_17_piece0 in memory on localhost:40577 (size: 2.4 KiB, free: 1920.0 MiB)
21:17:55.250 INFO  SparkContext - Created broadcast 17 from broadcast at DAGScheduler.scala:1580
21:17:55.252 INFO  DAGScheduler - Submitting 3 missing tasks from ResultStage 15 (MapPartitionsRDD[34] at mapPartitions at SparkSharder.java:386) (first 15 tasks are for partitions Vector(0, 1, 2))
21:17:55.252 INFO  TaskSchedulerImpl - Adding task set 15.0 with 3 tasks resource profile 0
21:17:55.253 INFO  TaskSetManager - Starting task 0.0 in stage 15.0 (TID 22) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7680 bytes) 
21:17:55.253 INFO  TaskSetManager - Starting task 1.0 in stage 15.0 (TID 23) (localhost, executor driver, partition 1, PROCESS_LOCAL, 7682 bytes) 
21:17:55.253 INFO  TaskSetManager - Starting task 2.0 in stage 15.0 (TID 24) (localhost, executor driver, partition 2, PROCESS_LOCAL, 7680 bytes) 
21:17:55.254 INFO  Executor - Running task 0.0 in stage 15.0 (TID 22)
21:17:55.255 INFO  Executor - Running task 2.0 in stage 15.0 (TID 24)
21:17:55.256 INFO  Executor - Running task 1.0 in stage 15.0 (TID 23)
21:17:55.267 INFO  Executor - Finished task 0.0 in stage 15.0 (TID 22). 1073 bytes result sent to driver
21:17:55.267 INFO  Executor - Finished task 1.0 in stage 15.0 (TID 23). 1073 bytes result sent to driver
21:17:55.268 INFO  Executor - Finished task 2.0 in stage 15.0 (TID 24). 1073 bytes result sent to driver
21:17:55.270 INFO  TaskSetManager - Finished task 0.0 in stage 15.0 (TID 22) in 18 ms on localhost (executor driver) (1/3)
21:17:55.271 INFO  TaskSetManager - Finished task 2.0 in stage 15.0 (TID 24) in 18 ms on localhost (executor driver) (2/3)
21:17:55.272 INFO  TaskSetManager - Finished task 1.0 in stage 15.0 (TID 23) in 19 ms on localhost (executor driver) (3/3)
21:17:55.272 INFO  TaskSchedulerImpl - Removed TaskSet 15.0, whose tasks have all completed, from pool 
21:17:55.273 INFO  DAGScheduler - ResultStage 15 (collect at SparkSharder.java:388) finished in 0.028 s
21:17:55.274 INFO  DAGScheduler - Job 11 is finished. Cancelling potential speculative or zombie tasks for this job
21:17:55.274 INFO  TaskSchedulerImpl - Killing all running tasks in stage 15: Stage finished
21:17:55.274 INFO  DAGScheduler - Job 11 finished: collect at SparkSharder.java:388, took 0.031365 s
21:17:55.282 INFO  SparkContext - Starting job: collect at SparkSharder.java:388
21:17:55.283 INFO  DAGScheduler - Got job 12 (collect at SparkSharder.java:388) with 1 output partitions
21:17:55.283 INFO  DAGScheduler - Final stage: ResultStage 16 (collect at SparkSharder.java:388)
21:17:55.283 INFO  DAGScheduler - Parents of final stage: List()
21:17:55.283 INFO  DAGScheduler - Missing parents: List()
21:17:55.284 INFO  DAGScheduler - Submitting ResultStage 16 (MapPartitionsRDD[36] at mapPartitions at SparkSharder.java:386), which has no missing parents
21:17:55.286 INFO  MemoryStore - Block broadcast_18 stored as values in memory (estimated size 4.5 KiB, free 1919.9 MiB)
21:17:55.288 INFO  MemoryStore - Block broadcast_18_piece0 stored as bytes in memory (estimated size 2.4 KiB, free 1919.9 MiB)
21:17:55.288 INFO  BlockManagerInfo - Added broadcast_18_piece0 in memory on localhost:40577 (size: 2.4 KiB, free: 1920.0 MiB)
21:17:55.288 INFO  SparkContext - Created broadcast 18 from broadcast at DAGScheduler.scala:1580
21:17:55.289 INFO  DAGScheduler - Submitting 1 missing tasks from ResultStage 16 (MapPartitionsRDD[36] at mapPartitions at SparkSharder.java:386) (first 15 tasks are for partitions Vector(0))
21:17:55.289 INFO  TaskSchedulerImpl - Adding task set 16.0 with 1 tasks resource profile 0
21:17:55.290 INFO  TaskSetManager - Starting task 0.0 in stage 16.0 (TID 25) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7730 bytes) 
21:17:55.290 INFO  Executor - Running task 0.0 in stage 16.0 (TID 25)
21:17:55.295 INFO  Executor - Finished task 0.0 in stage 16.0 (TID 25). 1030 bytes result sent to driver
21:17:55.296 INFO  TaskSetManager - Finished task 0.0 in stage 16.0 (TID 25) in 6 ms on localhost (executor driver) (1/1)
21:17:55.296 INFO  TaskSchedulerImpl - Removed TaskSet 16.0, whose tasks have all completed, from pool 
21:17:55.298 INFO  DAGScheduler - ResultStage 16 (collect at SparkSharder.java:388) finished in 0.014 s
21:17:55.298 INFO  DAGScheduler - Job 12 is finished. Cancelling potential speculative or zombie tasks for this job
21:17:55.298 INFO  TaskSchedulerImpl - Killing all running tasks in stage 16: Stage finished
21:17:55.300 INFO  DAGScheduler - Job 12 finished: collect at SparkSharder.java:388, took 0.017160 s
21:17:55.310 INFO  SparkContext - Starting job: collect at SparkSharder.java:388
21:17:55.311 INFO  DAGScheduler - Got job 13 (collect at SparkSharder.java:388) with 2 output partitions
21:17:55.311 INFO  DAGScheduler - Final stage: ResultStage 17 (collect at SparkSharder.java:388)
21:17:55.311 INFO  DAGScheduler - Parents of final stage: List()
21:17:55.311 INFO  DAGScheduler - Missing parents: List()
21:17:55.311 INFO  DAGScheduler - Submitting ResultStage 17 (MapPartitionsRDD[38] at mapPartitions at SparkSharder.java:386), which has no missing parents
21:17:55.314 INFO  MemoryStore - Block broadcast_19 stored as values in memory (estimated size 4.5 KiB, free 1919.9 MiB)
21:17:55.316 INFO  MemoryStore - Block broadcast_19_piece0 stored as bytes in memory (estimated size 2.4 KiB, free 1919.9 MiB)
21:17:55.317 INFO  BlockManagerInfo - Added broadcast_19_piece0 in memory on localhost:40577 (size: 2.4 KiB, free: 1919.9 MiB)
21:17:55.317 INFO  SparkContext - Created broadcast 19 from broadcast at DAGScheduler.scala:1580
21:17:55.317 INFO  DAGScheduler - Submitting 2 missing tasks from ResultStage 17 (MapPartitionsRDD[38] at mapPartitions at SparkSharder.java:386) (first 15 tasks are for partitions Vector(0, 1))
21:17:55.317 INFO  TaskSchedulerImpl - Adding task set 17.0 with 2 tasks resource profile 0
21:17:55.318 INFO  TaskSetManager - Starting task 0.0 in stage 17.0 (TID 26) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7692 bytes) 
21:17:55.319 INFO  TaskSetManager - Starting task 1.0 in stage 17.0 (TID 27) (localhost, executor driver, partition 1, PROCESS_LOCAL, 7692 bytes) 
21:17:55.319 INFO  Executor - Running task 0.0 in stage 17.0 (TID 26)
21:17:55.319 INFO  Executor - Running task 1.0 in stage 17.0 (TID 27)
21:17:55.322 INFO  Executor - Finished task 0.0 in stage 17.0 (TID 26). 1030 bytes result sent to driver
21:17:55.324 INFO  Executor - Finished task 1.0 in stage 17.0 (TID 27). 1030 bytes result sent to driver
21:17:55.327 INFO  TaskSetManager - Finished task 0.0 in stage 17.0 (TID 26) in 9 ms on localhost (executor driver) (1/2)
21:17:55.327 INFO  TaskSetManager - Finished task 1.0 in stage 17.0 (TID 27) in 9 ms on localhost (executor driver) (2/2)
21:17:55.327 INFO  TaskSchedulerImpl - Removed TaskSet 17.0, whose tasks have all completed, from pool 
21:17:55.328 INFO  DAGScheduler - ResultStage 17 (collect at SparkSharder.java:388) finished in 0.015 s
21:17:55.328 INFO  DAGScheduler - Job 13 is finished. Cancelling potential speculative or zombie tasks for this job
21:17:55.328 INFO  TaskSchedulerImpl - Killing all running tasks in stage 17: Stage finished
21:17:55.328 INFO  DAGScheduler - Job 13 finished: collect at SparkSharder.java:388, took 0.017585 s
21:17:55.337 INFO  SparkContext - Starting job: collect at SparkSharder.java:388
21:17:55.338 INFO  DAGScheduler - Got job 14 (collect at SparkSharder.java:388) with 3 output partitions
21:17:55.338 INFO  DAGScheduler - Final stage: ResultStage 18 (collect at SparkSharder.java:388)
21:17:55.338 INFO  DAGScheduler - Parents of final stage: List()
21:17:55.338 INFO  DAGScheduler - Missing parents: List()
21:17:55.338 INFO  DAGScheduler - Submitting ResultStage 18 (MapPartitionsRDD[40] at mapPartitions at SparkSharder.java:386), which has no missing parents
21:17:55.340 INFO  MemoryStore - Block broadcast_20 stored as values in memory (estimated size 4.5 KiB, free 1919.8 MiB)
21:17:55.340 INFO  MemoryStore - Block broadcast_20_piece0 stored as bytes in memory (estimated size 2.4 KiB, free 1919.8 MiB)
21:17:55.342 INFO  BlockManagerInfo - Added broadcast_20_piece0 in memory on localhost:40577 (size: 2.4 KiB, free: 1919.9 MiB)
21:17:55.342 INFO  SparkContext - Created broadcast 20 from broadcast at DAGScheduler.scala:1580
21:17:55.343 INFO  DAGScheduler - Submitting 3 missing tasks from ResultStage 18 (MapPartitionsRDD[40] at mapPartitions at SparkSharder.java:386) (first 15 tasks are for partitions Vector(0, 1, 2))
21:17:55.343 INFO  TaskSchedulerImpl - Adding task set 18.0 with 3 tasks resource profile 0
21:17:55.344 INFO  TaskSetManager - Starting task 0.0 in stage 18.0 (TID 28) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7680 bytes) 
21:17:55.344 INFO  TaskSetManager - Starting task 1.0 in stage 18.0 (TID 29) (localhost, executor driver, partition 1, PROCESS_LOCAL, 7682 bytes) 
21:17:55.345 INFO  TaskSetManager - Starting task 2.0 in stage 18.0 (TID 30) (localhost, executor driver, partition 2, PROCESS_LOCAL, 7680 bytes) 
21:17:55.346 INFO  Executor - Running task 2.0 in stage 18.0 (TID 30)
21:17:55.347 INFO  Executor - Running task 0.0 in stage 18.0 (TID 28)
21:17:55.349 INFO  Executor - Finished task 0.0 in stage 18.0 (TID 28). 1030 bytes result sent to driver
21:17:55.351 INFO  Executor - Running task 1.0 in stage 18.0 (TID 29)
21:17:55.351 INFO  TaskSetManager - Finished task 0.0 in stage 18.0 (TID 28) in 7 ms on localhost (executor driver) (1/3)
21:17:55.353 INFO  Executor - Finished task 1.0 in stage 18.0 (TID 29). 987 bytes result sent to driver
21:17:55.353 INFO  Executor - Finished task 2.0 in stage 18.0 (TID 30). 1030 bytes result sent to driver
21:17:55.357 INFO  TaskSetManager - Finished task 1.0 in stage 18.0 (TID 29) in 13 ms on localhost (executor driver) (2/3)
21:17:55.357 INFO  TaskSetManager - Finished task 2.0 in stage 18.0 (TID 30) in 12 ms on localhost (executor driver) (3/3)
21:17:55.357 INFO  TaskSchedulerImpl - Removed TaskSet 18.0, whose tasks have all completed, from pool 
21:17:55.358 INFO  DAGScheduler - ResultStage 18 (collect at SparkSharder.java:388) finished in 0.019 s
21:17:55.358 INFO  DAGScheduler - Job 14 is finished. Cancelling potential speculative or zombie tasks for this job
21:17:55.358 INFO  TaskSchedulerImpl - Killing all running tasks in stage 18: Stage finished
21:17:55.358 INFO  DAGScheduler - Job 14 finished: collect at SparkSharder.java:388, took 0.021154 s
21:17:55.373 INFO  SparkContext - Starting job: collect at SparkSharder.java:388
21:17:55.378 INFO  DAGScheduler - Got job 15 (collect at SparkSharder.java:388) with 4 output partitions
21:17:55.378 INFO  DAGScheduler - Final stage: ResultStage 19 (collect at SparkSharder.java:388)
21:17:55.378 INFO  DAGScheduler - Parents of final stage: List()
21:17:55.378 INFO  DAGScheduler - Missing parents: List()
21:17:55.378 INFO  DAGScheduler - Submitting ResultStage 19 (MapPartitionsRDD[43] at mapPartitions at SparkSharder.java:386), which has no missing parents
21:17:55.379 INFO  MemoryStore - Block broadcast_21 stored as values in memory (estimated size 4.5 KiB, free 1919.8 MiB)
21:17:55.380 INFO  MemoryStore - Block broadcast_21_piece0 stored as bytes in memory (estimated size 2.4 KiB, free 1919.8 MiB)
21:17:55.382 INFO  BlockManagerInfo - Added broadcast_21_piece0 in memory on localhost:40577 (size: 2.4 KiB, free: 1919.9 MiB)
21:17:55.382 INFO  SparkContext - Created broadcast 21 from broadcast at DAGScheduler.scala:1580
21:17:55.382 INFO  DAGScheduler - Submitting 4 missing tasks from ResultStage 19 (MapPartitionsRDD[43] at mapPartitions at SparkSharder.java:386) (first 15 tasks are for partitions Vector(0, 1, 2, 3))
21:17:55.382 INFO  TaskSchedulerImpl - Adding task set 19.0 with 4 tasks resource profile 0
21:17:55.384 INFO  TaskSetManager - Starting task 0.0 in stage 19.0 (TID 31) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7692 bytes) 
21:17:55.384 INFO  TaskSetManager - Starting task 1.0 in stage 19.0 (TID 32) (localhost, executor driver, partition 1, PROCESS_LOCAL, 7692 bytes) 
21:17:55.384 INFO  TaskSetManager - Starting task 2.0 in stage 19.0 (TID 33) (localhost, executor driver, partition 2, PROCESS_LOCAL, 7692 bytes) 
21:17:55.385 INFO  TaskSetManager - Starting task 3.0 in stage 19.0 (TID 34) (localhost, executor driver, partition 3, PROCESS_LOCAL, 7692 bytes) 
21:17:55.385 INFO  Executor - Running task 0.0 in stage 19.0 (TID 31)
21:17:55.385 INFO  Executor - Running task 3.0 in stage 19.0 (TID 34)
21:17:55.386 INFO  Executor - Running task 1.0 in stage 19.0 (TID 32)
21:17:55.388 INFO  Executor - Running task 2.0 in stage 19.0 (TID 33)
21:17:55.391 INFO  Executor - Finished task 3.0 in stage 19.0 (TID 34). 1073 bytes result sent to driver
21:17:55.405 INFO  TaskSetManager - Finished task 3.0 in stage 19.0 (TID 34) in 21 ms on localhost (executor driver) (1/4)
21:17:55.409 INFO  Executor - Finished task 0.0 in stage 19.0 (TID 31). 1073 bytes result sent to driver
21:17:55.410 INFO  TaskSetManager - Finished task 0.0 in stage 19.0 (TID 31) in 27 ms on localhost (executor driver) (2/4)
21:17:55.414 INFO  Executor - Finished task 1.0 in stage 19.0 (TID 32). 1030 bytes result sent to driver
21:17:55.415 INFO  Executor - Finished task 2.0 in stage 19.0 (TID 33). 1030 bytes result sent to driver
21:17:55.416 INFO  TaskSetManager - Finished task 1.0 in stage 19.0 (TID 32) in 32 ms on localhost (executor driver) (3/4)
21:17:55.418 INFO  TaskSetManager - Finished task 2.0 in stage 19.0 (TID 33) in 34 ms on localhost (executor driver) (4/4)
21:17:55.418 INFO  TaskSchedulerImpl - Removed TaskSet 19.0, whose tasks have all completed, from pool 
21:17:55.418 INFO  DAGScheduler - ResultStage 19 (collect at SparkSharder.java:388) finished in 0.040 s
21:17:55.418 INFO  DAGScheduler - Job 15 is finished. Cancelling potential speculative or zombie tasks for this job
21:17:55.418 INFO  TaskSchedulerImpl - Killing all running tasks in stage 19: Stage finished
21:17:55.420 INFO  DAGScheduler - Job 15 finished: collect at SparkSharder.java:388, took 0.042517 s
21:17:55.421 INFO  MemoryStore - Block broadcast_22 stored as values in memory (estimated size 464.0 B, free 1919.8 MiB)
21:17:55.422 INFO  MemoryStore - Block broadcast_22_piece0 stored as bytes in memory (estimated size 159.0 B, free 1919.8 MiB)
21:17:55.422 INFO  BlockManagerInfo - Added broadcast_22_piece0 in memory on localhost:40577 (size: 159.0 B, free: 1919.9 MiB)
21:17:55.425 INFO  SparkContext - Created broadcast 22 from broadcast at SparkSharder.java:214
21:17:55.426 INFO  MemoryStore - Block broadcast_23 stored as values in memory (estimated size 1088.0 B, free 1919.8 MiB)
21:17:55.427 INFO  MemoryStore - Block broadcast_23_piece0 stored as bytes in memory (estimated size 465.0 B, free 1919.8 MiB)
21:17:55.427 INFO  BlockManagerInfo - Added broadcast_23_piece0 in memory on localhost:40577 (size: 465.0 B, free: 1919.9 MiB)
21:17:55.428 INFO  SparkContext - Created broadcast 23 from broadcast at SparkSharder.java:221
21:17:55.454 INFO  SparkContext - Starting job: collectAsMap at SparkSharder.java:258
21:17:55.456 INFO  DAGScheduler - Registering RDD 45 (mapToPair at SparkSharder.java:247) as input to shuffle 4
21:17:55.457 INFO  DAGScheduler - Registering RDD 48 (mapToPair at SparkSharder.java:255) as input to shuffle 3
21:17:55.457 INFO  DAGScheduler - Got job 16 (collectAsMap at SparkSharder.java:258) with 4 output partitions
21:17:55.457 INFO  DAGScheduler - Final stage: ResultStage 22 (collectAsMap at SparkSharder.java:258)
21:17:55.457 INFO  DAGScheduler - Parents of final stage: List(ShuffleMapStage 21)
21:17:55.457 INFO  DAGScheduler - Missing parents: List(ShuffleMapStage 21)
21:17:55.467 INFO  DAGScheduler - Submitting ShuffleMapStage 20 (MapPartitionsRDD[45] at mapToPair at SparkSharder.java:247), which has no missing parents
21:17:55.469 INFO  MemoryStore - Block broadcast_24 stored as values in memory (estimated size 7.9 KiB, free 1919.8 MiB)
21:17:55.469 INFO  MemoryStore - Block broadcast_24_piece0 stored as bytes in memory (estimated size 4.3 KiB, free 1919.8 MiB)
21:17:55.470 INFO  BlockManagerInfo - Added broadcast_24_piece0 in memory on localhost:40577 (size: 4.3 KiB, free: 1919.9 MiB)
21:17:55.470 INFO  SparkContext - Created broadcast 24 from broadcast at DAGScheduler.scala:1580
21:17:55.471 INFO  DAGScheduler - Submitting 4 missing tasks from ShuffleMapStage 20 (MapPartitionsRDD[45] at mapToPair at SparkSharder.java:247) (first 15 tasks are for partitions Vector(0, 1, 2, 3))
21:17:55.471 INFO  TaskSchedulerImpl - Adding task set 20.0 with 4 tasks resource profile 0
21:17:55.474 INFO  TaskSetManager - Starting task 0.0 in stage 20.0 (TID 35) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7569 bytes) 
21:17:55.475 INFO  TaskSetManager - Starting task 1.0 in stage 20.0 (TID 36) (localhost, executor driver, partition 1, PROCESS_LOCAL, 7636 bytes) 
21:17:55.475 INFO  TaskSetManager - Starting task 2.0 in stage 20.0 (TID 37) (localhost, executor driver, partition 2, PROCESS_LOCAL, 7636 bytes) 
21:17:55.475 INFO  TaskSetManager - Starting task 3.0 in stage 20.0 (TID 38) (localhost, executor driver, partition 3, PROCESS_LOCAL, 7636 bytes) 
21:17:55.476 INFO  Executor - Running task 3.0 in stage 20.0 (TID 38)
21:17:55.477 INFO  Executor - Running task 1.0 in stage 20.0 (TID 36)
21:17:55.477 INFO  Executor - Running task 2.0 in stage 20.0 (TID 37)
21:17:55.478 INFO  BlockManagerInfo - Removed broadcast_6_piece0 on localhost:40577 in memory (size: 5.0 KiB, free: 1919.9 MiB)
21:17:55.480 INFO  Executor - Running task 0.0 in stage 20.0 (TID 35)
21:17:55.491 INFO  Executor - Finished task 0.0 in stage 20.0 (TID 35). 936 bytes result sent to driver
21:17:55.492 INFO  TaskSetManager - Finished task 0.0 in stage 20.0 (TID 35) in 18 ms on localhost (executor driver) (1/4)
21:17:55.492 INFO  Executor - Finished task 1.0 in stage 20.0 (TID 36). 1065 bytes result sent to driver
21:17:55.493 INFO  TaskSetManager - Finished task 1.0 in stage 20.0 (TID 36) in 18 ms on localhost (executor driver) (2/4)
21:17:55.494 INFO  Executor - Finished task 3.0 in stage 20.0 (TID 38). 1065 bytes result sent to driver
21:17:55.494 INFO  TaskSetManager - Finished task 3.0 in stage 20.0 (TID 38) in 19 ms on localhost (executor driver) (3/4)
21:17:55.500 INFO  BlockManagerInfo - Removed broadcast_13_piece0 on localhost:40577 in memory (size: 2.4 KiB, free: 1919.9 MiB)
21:17:55.502 INFO  Executor - Finished task 2.0 in stage 20.0 (TID 37). 1065 bytes result sent to driver
21:17:55.503 INFO  TaskSetManager - Finished task 2.0 in stage 20.0 (TID 37) in 27 ms on localhost (executor driver) (4/4)
21:17:55.503 INFO  TaskSchedulerImpl - Removed TaskSet 20.0, whose tasks have all completed, from pool 
21:17:55.503 INFO  DAGScheduler - ShuffleMapStage 20 (mapToPair at SparkSharder.java:247) finished in 0.036 s
21:17:55.504 INFO  DAGScheduler - looking for newly runnable stages
21:17:55.504 INFO  DAGScheduler - running: HashSet()
21:17:55.504 INFO  DAGScheduler - waiting: HashSet(ShuffleMapStage 21, ResultStage 22)
21:17:55.504 INFO  DAGScheduler - failed: HashSet()
21:17:55.505 INFO  DAGScheduler - Submitting ShuffleMapStage 21 (MapPartitionsRDD[48] at mapToPair at SparkSharder.java:255), which has no missing parents
21:17:55.507 INFO  MemoryStore - Block broadcast_25 stored as values in memory (estimated size 8.2 KiB, free 1919.8 MiB)
21:17:55.508 INFO  MemoryStore - Block broadcast_25_piece0 stored as bytes in memory (estimated size 4.5 KiB, free 1919.8 MiB)
21:17:55.508 INFO  BlockManagerInfo - Added broadcast_25_piece0 in memory on localhost:40577 (size: 4.5 KiB, free: 1919.9 MiB)
21:17:55.509 INFO  SparkContext - Created broadcast 25 from broadcast at DAGScheduler.scala:1580
21:17:55.509 INFO  DAGScheduler - Submitting 4 missing tasks from ShuffleMapStage 21 (MapPartitionsRDD[48] at mapToPair at SparkSharder.java:255) (first 15 tasks are for partitions Vector(0, 1, 2, 3))
21:17:55.509 INFO  TaskSchedulerImpl - Adding task set 21.0 with 4 tasks resource profile 0
21:17:55.511 INFO  TaskSetManager - Starting task 0.0 in stage 21.0 (TID 39) (localhost, executor driver, partition 0, NODE_LOCAL, 7502 bytes) 
21:17:55.511 INFO  TaskSetManager - Starting task 2.0 in stage 21.0 (TID 40) (localhost, executor driver, partition 2, NODE_LOCAL, 7502 bytes) 
21:17:55.512 INFO  TaskSetManager - Starting task 1.0 in stage 21.0 (TID 41) (localhost, executor driver, partition 1, PROCESS_LOCAL, 7502 bytes) 
21:17:55.512 INFO  TaskSetManager - Starting task 3.0 in stage 21.0 (TID 42) (localhost, executor driver, partition 3, PROCESS_LOCAL, 7502 bytes) 
21:17:55.512 INFO  Executor - Running task 3.0 in stage 21.0 (TID 42)
21:17:55.513 INFO  Executor - Running task 0.0 in stage 21.0 (TID 39)
21:17:55.514 INFO  Executor - Running task 1.0 in stage 21.0 (TID 41)
21:17:55.515 INFO  Executor - Running task 2.0 in stage 21.0 (TID 40)
21:17:55.519 INFO  ShuffleBlockFetcherIterator - Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
21:17:55.519 INFO  ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
21:17:55.520 INFO  MemoryStore - Block rdd_47_3 stored as values in memory (estimated size 16.0 B, free 1919.8 MiB)
21:17:55.521 INFO  ShuffleBlockFetcherIterator - Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
21:17:55.522 INFO  ShuffleBlockFetcherIterator - Started 0 remote fetches in 3 ms
21:17:55.522 INFO  MemoryStore - Block rdd_47_1 stored as values in memory (estimated size 16.0 B, free 1919.8 MiB)
21:17:55.523 INFO  BlockManagerInfo - Added rdd_47_3 in memory on localhost:40577 (size: 16.0 B, free: 1919.9 MiB)
21:17:55.523 INFO  BlockManagerInfo - Removed broadcast_14_piece0 on localhost:40577 in memory (size: 2.4 KiB, free: 1919.9 MiB)
21:17:55.523 INFO  BlockManagerInfo - Added rdd_47_1 in memory on localhost:40577 (size: 16.0 B, free: 1919.9 MiB)
21:17:55.524 INFO  ShuffleBlockFetcherIterator - Getting 1 (171.0 B) non-empty blocks including 1 (171.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
21:17:55.527 INFO  Executor - Finished task 3.0 in stage 21.0 (TID 42). 1839 bytes result sent to driver
21:17:55.527 INFO  ShuffleBlockFetcherIterator - Started 0 remote fetches in 3 ms
21:17:55.524 INFO  ShuffleBlockFetcherIterator - Getting 2 (342.0 B) non-empty blocks including 2 (342.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
21:17:55.529 INFO  ShuffleBlockFetcherIterator - Started 0 remote fetches in 4 ms
21:17:55.530 INFO  BlockManagerInfo - Removed broadcast_12_piece0 on localhost:40577 in memory (size: 2.4 KiB, free: 1919.9 MiB)
21:17:55.531 INFO  TaskSetManager - Finished task 3.0 in stage 21.0 (TID 42) in 19 ms on localhost (executor driver) (1/4)
21:17:55.526 INFO  Executor - Finished task 1.0 in stage 21.0 (TID 41). 1839 bytes result sent to driver
21:17:55.535 INFO  TaskSetManager - Finished task 1.0 in stage 21.0 (TID 41) in 23 ms on localhost (executor driver) (2/4)
21:17:55.539 INFO  MemoryStore - Block rdd_47_2 stored as values in memory (estimated size 144.0 B, free 1919.8 MiB)
21:17:55.539 INFO  BlockManagerInfo - Added rdd_47_2 in memory on localhost:40577 (size: 144.0 B, free: 1919.9 MiB)
21:17:55.541 INFO  MemoryStore - Block rdd_47_0 stored as values in memory (estimated size 264.0 B, free 1919.8 MiB)
21:17:55.544 INFO  BlockManagerInfo - Added rdd_47_0 in memory on localhost:40577 (size: 264.0 B, free: 1919.9 MiB)
21:17:55.545 INFO  BlockManagerInfo - Removed broadcast_16_piece0 on localhost:40577 in memory (size: 2.4 KiB, free: 1919.9 MiB)
21:17:55.548 INFO  Executor - Finished task 2.0 in stage 21.0 (TID 40). 1925 bytes result sent to driver
21:17:55.549 INFO  TaskSetManager - Finished task 2.0 in stage 21.0 (TID 40) in 38 ms on localhost (executor driver) (3/4)
21:17:55.551 INFO  Executor - Finished task 0.0 in stage 21.0 (TID 39). 1925 bytes result sent to driver
21:17:55.552 INFO  TaskSetManager - Finished task 0.0 in stage 21.0 (TID 39) in 41 ms on localhost (executor driver) (4/4)
21:17:55.554 INFO  TaskSchedulerImpl - Removed TaskSet 21.0, whose tasks have all completed, from pool 
21:17:55.554 INFO  DAGScheduler - ShuffleMapStage 21 (mapToPair at SparkSharder.java:255) finished in 0.049 s
21:17:55.554 INFO  DAGScheduler - looking for newly runnable stages
21:17:55.554 INFO  DAGScheduler - running: HashSet()
21:17:55.554 INFO  DAGScheduler - waiting: HashSet(ResultStage 22)
21:17:55.554 INFO  DAGScheduler - failed: HashSet()
21:17:55.554 INFO  DAGScheduler - Submitting ResultStage 22 (ShuffledRDD[49] at reduceByKey at SparkSharder.java:257), which has no missing parents
21:17:55.556 INFO  MemoryStore - Block broadcast_26 stored as values in memory (estimated size 5.8 KiB, free 1919.8 MiB)
21:17:55.558 INFO  MemoryStore - Block broadcast_26_piece0 stored as bytes in memory (estimated size 3.2 KiB, free 1919.8 MiB)
21:17:55.558 INFO  BlockManagerInfo - Removed broadcast_17_piece0 on localhost:40577 in memory (size: 2.4 KiB, free: 1920.0 MiB)
21:17:55.559 INFO  BlockManagerInfo - Added broadcast_26_piece0 in memory on localhost:40577 (size: 3.2 KiB, free: 1919.9 MiB)
21:17:55.560 INFO  SparkContext - Created broadcast 26 from broadcast at DAGScheduler.scala:1580
21:17:55.560 INFO  DAGScheduler - Submitting 4 missing tasks from ResultStage 22 (ShuffledRDD[49] at reduceByKey at SparkSharder.java:257) (first 15 tasks are for partitions Vector(0, 1, 2, 3))
21:17:55.560 INFO  TaskSchedulerImpl - Adding task set 22.0 with 4 tasks resource profile 0
21:17:55.561 INFO  TaskSetManager - Starting task 0.0 in stage 22.0 (TID 43) (localhost, executor driver, partition 0, NODE_LOCAL, 7513 bytes) 
21:17:55.561 INFO  TaskSetManager - Starting task 2.0 in stage 22.0 (TID 44) (localhost, executor driver, partition 2, NODE_LOCAL, 7513 bytes) 
21:17:55.562 INFO  TaskSetManager - Starting task 1.0 in stage 22.0 (TID 45) (localhost, executor driver, partition 1, PROCESS_LOCAL, 7513 bytes) 
21:17:55.562 INFO  TaskSetManager - Starting task 3.0 in stage 22.0 (TID 46) (localhost, executor driver, partition 3, PROCESS_LOCAL, 7513 bytes) 
21:17:55.563 INFO  Executor - Running task 3.0 in stage 22.0 (TID 46)
21:17:55.563 INFO  Executor - Running task 1.0 in stage 22.0 (TID 45)
21:17:55.563 INFO  Executor - Running task 0.0 in stage 22.0 (TID 43)
21:17:55.567 INFO  ShuffleBlockFetcherIterator - Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
21:17:55.569 INFO  ShuffleBlockFetcherIterator - Started 0 remote fetches in 2 ms
21:17:55.573 INFO  Executor - Finished task 3.0 in stage 22.0 (TID 46). 1758 bytes result sent to driver
21:17:55.568 INFO  ShuffleBlockFetcherIterator - Getting 1 (54.0 B) non-empty blocks including 1 (54.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
21:17:55.574 INFO  ShuffleBlockFetcherIterator - Started 0 remote fetches in 6 ms
21:17:55.567 INFO  ShuffleBlockFetcherIterator - Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
21:17:55.577 INFO  ShuffleBlockFetcherIterator - Started 0 remote fetches in 9 ms
21:17:55.578 INFO  Executor - Finished task 1.0 in stage 22.0 (TID 45). 1758 bytes result sent to driver
21:17:55.578 INFO  TaskSetManager - Finished task 3.0 in stage 22.0 (TID 46) in 16 ms on localhost (executor driver) (1/4)
21:17:55.567 INFO  Executor - Running task 2.0 in stage 22.0 (TID 44)
21:17:55.578 INFO  TaskSetManager - Finished task 1.0 in stage 22.0 (TID 45) in 16 ms on localhost (executor driver) (2/4)
21:17:55.571 INFO  BlockManagerInfo - Removed broadcast_21_piece0 on localhost:40577 in memory (size: 2.4 KiB, free: 1919.9 MiB)
21:17:55.584 INFO  ShuffleBlockFetcherIterator - Getting 1 (54.0 B) non-empty blocks including 1 (54.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
21:17:55.584 INFO  ShuffleBlockFetcherIterator - Started 0 remote fetches in 3 ms
21:17:55.588 INFO  Executor - Finished task 0.0 in stage 22.0 (TID 43). 1770 bytes result sent to driver
21:17:55.589 INFO  TaskSetManager - Finished task 0.0 in stage 22.0 (TID 43) in 27 ms on localhost (executor driver) (3/4)
21:17:55.592 INFO  Executor - Finished task 2.0 in stage 22.0 (TID 44). 1770 bytes result sent to driver
21:17:55.595 INFO  TaskSetManager - Finished task 2.0 in stage 22.0 (TID 44) in 34 ms on localhost (executor driver) (4/4)
21:17:55.595 INFO  TaskSchedulerImpl - Removed TaskSet 22.0, whose tasks have all completed, from pool 
21:17:55.595 INFO  DAGScheduler - ResultStage 22 (collectAsMap at SparkSharder.java:258) finished in 0.040 s
21:17:55.596 INFO  DAGScheduler - Job 16 is finished. Cancelling potential speculative or zombie tasks for this job
21:17:55.596 INFO  TaskSchedulerImpl - Killing all running tasks in stage 22: Stage finished
21:17:55.596 INFO  DAGScheduler - Job 16 finished: collectAsMap at SparkSharder.java:258, took 0.140949 s
21:17:55.602 INFO  BlockManagerInfo - Removed broadcast_18_piece0 on localhost:40577 in memory (size: 2.4 KiB, free: 1920.0 MiB)
21:17:55.611 INFO  SparkContext - Starting job: collectAsMap at SparkSharderUnitTest.java:244
21:17:55.612 INFO  DAGScheduler - Got job 17 (collectAsMap at SparkSharderUnitTest.java:244) with 4 output partitions
21:17:55.612 INFO  DAGScheduler - Final stage: ResultStage 24 (collectAsMap at SparkSharderUnitTest.java:244)
21:17:55.612 INFO  DAGScheduler - Parents of final stage: List(ShuffleMapStage 23)
21:17:55.613 INFO  DAGScheduler - Missing parents: List()
21:17:55.613 INFO  DAGScheduler - Submitting ResultStage 24 (MapPartitionsRDD[53] at flatMapToPair at SparkSharderUnitTest.java:243), which has no missing parents
21:17:55.614 INFO  BlockManagerInfo - Removed broadcast_2_piece0 on localhost:40577 in memory (size: 349.0 B, free: 1920.0 MiB)
21:17:55.615 INFO  MemoryStore - Block broadcast_27 stored as values in memory (estimated size 10.3 KiB, free 1919.9 MiB)
21:17:55.616 INFO  MemoryStore - Block broadcast_27_piece0 stored as bytes in memory (estimated size 5.1 KiB, free 1919.8 MiB)
21:17:55.616 INFO  BlockManagerInfo - Added broadcast_27_piece0 in memory on localhost:40577 (size: 5.1 KiB, free: 1919.9 MiB)
21:17:55.616 INFO  SparkContext - Created broadcast 27 from broadcast at DAGScheduler.scala:1580
21:17:55.617 INFO  DAGScheduler - Submitting 4 missing tasks from ResultStage 24 (MapPartitionsRDD[53] at flatMapToPair at SparkSharderUnitTest.java:243) (first 15 tasks are for partitions Vector(0, 1, 2, 3))
21:17:55.617 INFO  TaskSchedulerImpl - Adding task set 24.0 with 4 tasks resource profile 0
21:17:55.618 INFO  TaskSetManager - Starting task 0.0 in stage 24.0 (TID 47) (localhost, executor driver, partition 0, PROCESS_LOCAL, 8821 bytes) 
21:17:55.618 INFO  TaskSetManager - Starting task 1.0 in stage 24.0 (TID 48) (localhost, executor driver, partition 1, PROCESS_LOCAL, 8533 bytes) 
21:17:55.619 INFO  TaskSetManager - Starting task 2.0 in stage 24.0 (TID 49) (localhost, executor driver, partition 2, PROCESS_LOCAL, 8677 bytes) 
21:17:55.619 INFO  TaskSetManager - Starting task 3.0 in stage 24.0 (TID 50) (localhost, executor driver, partition 3, PROCESS_LOCAL, 8533 bytes) 
21:17:55.620 INFO  Executor - Running task 0.0 in stage 24.0 (TID 47)
21:17:55.622 INFO  Executor - Running task 1.0 in stage 24.0 (TID 48)
21:17:55.624 INFO  BlockManager - Found block rdd_47_0 locally
21:17:55.626 INFO  Executor - Finished task 0.0 in stage 24.0 (TID 47). 1076 bytes result sent to driver
21:17:55.626 INFO  BlockManager - Found block rdd_47_1 locally
21:17:55.626 INFO  BlockManagerInfo - Removed broadcast_8_piece0 on localhost:40577 in memory (size: 3.8 KiB, free: 1920.0 MiB)
21:17:55.627 INFO  Executor - Running task 2.0 in stage 24.0 (TID 49)
21:17:55.629 INFO  BlockManager - Found block rdd_47_2 locally
21:17:55.629 INFO  Executor - Finished task 1.0 in stage 24.0 (TID 48). 941 bytes result sent to driver
21:17:55.630 INFO  Executor - Finished task 2.0 in stage 24.0 (TID 49). 1012 bytes result sent to driver
21:17:55.630 INFO  Executor - Running task 3.0 in stage 24.0 (TID 50)
21:17:55.630 INFO  TaskSetManager - Finished task 1.0 in stage 24.0 (TID 48) in 12 ms on localhost (executor driver) (1/4)
21:17:55.631 INFO  TaskSetManager - Finished task 2.0 in stage 24.0 (TID 49) in 13 ms on localhost (executor driver) (2/4)
21:17:55.631 INFO  TaskSetManager - Finished task 0.0 in stage 24.0 (TID 47) in 14 ms on localhost (executor driver) (3/4)
21:17:55.632 INFO  BlockManager - Found block rdd_47_3 locally
21:17:55.636 INFO  Executor - Finished task 3.0 in stage 24.0 (TID 50). 898 bytes result sent to driver
21:17:55.636 INFO  TaskSetManager - Finished task 3.0 in stage 24.0 (TID 50) in 17 ms on localhost (executor driver) (4/4)
21:17:55.636 INFO  TaskSchedulerImpl - Removed TaskSet 24.0, whose tasks have all completed, from pool 
21:17:55.637 INFO  DAGScheduler - ResultStage 24 (collectAsMap at SparkSharderUnitTest.java:244) finished in 0.023 s
21:17:55.640 INFO  DAGScheduler - Job 17 is finished. Cancelling potential speculative or zombie tasks for this job
21:17:55.640 INFO  TaskSchedulerImpl - Killing all running tasks in stage 24: Stage finished
21:17:55.640 INFO  DAGScheduler - Job 17 finished: collectAsMap at SparkSharderUnitTest.java:244, took 0.029465 s
21:17:55.644 INFO  MemoryStore - Block broadcast_28 stored as values in memory (estimated size 616.0 B, free 1919.9 MiB)
21:17:55.645 INFO  BlockManager - Removing RDD 7
21:17:55.646 INFO  MemoryStore - Block broadcast_28_piece0 stored as bytes in memory (estimated size 320.0 B, free 1919.9 MiB)
21:17:55.646 INFO  BlockManagerInfo - Added broadcast_28_piece0 in memory on localhost:40577 (size: 320.0 B, free: 1920.0 MiB)
21:17:55.647 INFO  SparkContext - Created broadcast 28 from broadcast at SparkSharder.java:100
21:17:55.659 INFO  BlockManagerInfo - Removed broadcast_1_piece0 on localhost:40577 in memory (size: 143.0 B, free: 1920.0 MiB)
21:17:55.667 INFO  BlockManagerInfo - Removed broadcast_15_piece0 on localhost:40577 in memory (size: 2.4 KiB, free: 1920.0 MiB)
21:17:55.669 INFO  SparkContext - Starting job: collectAsMap at SparkSharderUnitTest.java:249
21:17:55.670 INFO  DAGScheduler - Registering RDD 54 (flatMapToPair at SparkSharder.java:101) as input to shuffle 5
21:17:55.670 INFO  DAGScheduler - Got job 18 (collectAsMap at SparkSharderUnitTest.java:249) with 4 output partitions
21:17:55.670 INFO  DAGScheduler - Final stage: ResultStage 26 (collectAsMap at SparkSharderUnitTest.java:249)
21:17:55.670 INFO  DAGScheduler - Parents of final stage: List(ShuffleMapStage 25)
21:17:55.670 INFO  DAGScheduler - Missing parents: List(ShuffleMapStage 25)
21:17:55.670 INFO  DAGScheduler - Submitting ShuffleMapStage 25 (MapPartitionsRDD[54] at flatMapToPair at SparkSharder.java:101), which has no missing parents
21:17:55.672 INFO  MemoryStore - Block broadcast_29 stored as values in memory (estimated size 7.2 KiB, free 1919.9 MiB)
21:17:55.676 INFO  MemoryStore - Block broadcast_29_piece0 stored as bytes in memory (estimated size 3.8 KiB, free 1919.9 MiB)
21:17:55.677 INFO  BlockManagerInfo - Added broadcast_29_piece0 in memory on localhost:40577 (size: 3.8 KiB, free: 1919.9 MiB)
21:17:55.679 INFO  SparkContext - Created broadcast 29 from broadcast at DAGScheduler.scala:1580
21:17:55.679 INFO  DAGScheduler - Submitting 4 missing tasks from ShuffleMapStage 25 (MapPartitionsRDD[54] at flatMapToPair at SparkSharder.java:101) (first 15 tasks are for partitions Vector(0, 1, 2, 3))
21:17:55.679 INFO  TaskSchedulerImpl - Adding task set 25.0 with 4 tasks resource profile 0
21:17:55.681 INFO  TaskSetManager - Starting task 0.0 in stage 25.0 (TID 51) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7681 bytes) 
21:17:55.681 INFO  TaskSetManager - Starting task 1.0 in stage 25.0 (TID 52) (localhost, executor driver, partition 1, PROCESS_LOCAL, 7681 bytes) 
21:17:55.681 INFO  TaskSetManager - Starting task 2.0 in stage 25.0 (TID 53) (localhost, executor driver, partition 2, PROCESS_LOCAL, 7681 bytes) 
21:17:55.684 INFO  TaskSetManager - Starting task 3.0 in stage 25.0 (TID 54) (localhost, executor driver, partition 3, PROCESS_LOCAL, 7681 bytes) 
21:17:55.685 INFO  Executor - Running task 3.0 in stage 25.0 (TID 54)
21:17:55.690 INFO  Executor - Running task 0.0 in stage 25.0 (TID 51)
21:17:55.692 INFO  Executor - Running task 1.0 in stage 25.0 (TID 52)
21:17:55.696 INFO  Executor - Running task 2.0 in stage 25.0 (TID 53)
21:17:55.699 INFO  BlockManagerInfo - Removed broadcast_10_piece0 on localhost:40577 in memory (size: 2.4 KiB, free: 1920.0 MiB)
21:17:55.719 INFO  BlockManagerInfo - Removed broadcast_5_piece0 on localhost:40577 in memory (size: 3.2 KiB, free: 1920.0 MiB)
21:17:55.721 INFO  Executor - Finished task 1.0 in stage 25.0 (TID 52). 1065 bytes result sent to driver
21:17:55.724 INFO  TaskSetManager - Finished task 1.0 in stage 25.0 (TID 52) in 43 ms on localhost (executor driver) (1/4)
21:17:55.713 INFO  Executor - Finished task 3.0 in stage 25.0 (TID 54). 1065 bytes result sent to driver
21:17:55.727 INFO  Executor - Finished task 2.0 in stage 25.0 (TID 53). 1108 bytes result sent to driver
21:17:55.727 INFO  TaskSetManager - Finished task 3.0 in stage 25.0 (TID 54) in 46 ms on localhost (executor driver) (2/4)
21:17:55.728 INFO  TaskSetManager - Finished task 2.0 in stage 25.0 (TID 53) in 47 ms on localhost (executor driver) (3/4)
21:17:55.729 INFO  BlockManagerInfo - Removed broadcast_20_piece0 on localhost:40577 in memory (size: 2.4 KiB, free: 1920.0 MiB)
21:17:55.731 INFO  Executor - Finished task 0.0 in stage 25.0 (TID 51). 1065 bytes result sent to driver
21:17:55.733 INFO  TaskSetManager - Finished task 0.0 in stage 25.0 (TID 51) in 53 ms on localhost (executor driver) (4/4)
21:17:55.733 INFO  TaskSchedulerImpl - Removed TaskSet 25.0, whose tasks have all completed, from pool 
21:17:55.734 INFO  DAGScheduler - ShuffleMapStage 25 (flatMapToPair at SparkSharder.java:101) finished in 0.063 s
21:17:55.734 INFO  DAGScheduler - looking for newly runnable stages
21:17:55.734 INFO  DAGScheduler - running: HashSet()
21:17:55.734 INFO  DAGScheduler - waiting: HashSet(ResultStage 26)
21:17:55.734 INFO  DAGScheduler - failed: HashSet()
21:17:55.735 INFO  DAGScheduler - Submitting ResultStage 26 (MapPartitionsRDD[58] at flatMapToPair at SparkSharderUnitTest.java:248), which has no missing parents
21:17:55.736 INFO  MemoryStore - Block broadcast_30 stored as values in memory (estimated size 9.4 KiB, free 1919.9 MiB)
21:17:55.740 INFO  MemoryStore - Block broadcast_30_piece0 stored as bytes in memory (estimated size 4.7 KiB, free 1919.9 MiB)
21:17:55.740 INFO  BlockManagerInfo - Added broadcast_30_piece0 in memory on localhost:40577 (size: 4.7 KiB, free: 1920.0 MiB)
21:17:55.741 INFO  SparkContext - Created broadcast 30 from broadcast at DAGScheduler.scala:1580
21:17:55.741 INFO  DAGScheduler - Submitting 4 missing tasks from ResultStage 26 (MapPartitionsRDD[58] at flatMapToPair at SparkSharderUnitTest.java:248) (first 15 tasks are for partitions Vector(0, 1, 2, 3))
21:17:55.741 INFO  TaskSchedulerImpl - Adding task set 26.0 with 4 tasks resource profile 0
21:17:55.742 INFO  TaskSetManager - Starting task 1.0 in stage 26.0 (TID 55) (localhost, executor driver, partition 1, NODE_LOCAL, 7513 bytes) 
21:17:55.742 INFO  TaskSetManager - Starting task 0.0 in stage 26.0 (TID 56) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7513 bytes) 
21:17:55.743 INFO  TaskSetManager - Starting task 2.0 in stage 26.0 (TID 57) (localhost, executor driver, partition 2, PROCESS_LOCAL, 7513 bytes) 
21:17:55.743 INFO  TaskSetManager - Starting task 3.0 in stage 26.0 (TID 58) (localhost, executor driver, partition 3, PROCESS_LOCAL, 7513 bytes) 
21:17:55.743 INFO  Executor - Running task 3.0 in stage 26.0 (TID 58)
21:17:55.743 INFO  Executor - Running task 2.0 in stage 26.0 (TID 57)
21:17:55.743 INFO  Executor - Running task 0.0 in stage 26.0 (TID 56)
21:17:55.744 INFO  Executor - Running task 1.0 in stage 26.0 (TID 55)
21:17:55.744 INFO  BlockManagerInfo - Removed broadcast_11_piece0 on localhost:40577 in memory (size: 2.4 KiB, free: 1920.0 MiB)
21:17:55.750 INFO  ShuffleBlockFetcherIterator - Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
21:17:55.750 INFO  ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
21:17:55.752 INFO  ShuffleBlockFetcherIterator - Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
21:17:55.752 INFO  ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
21:17:55.754 INFO  ShuffleBlockFetcherIterator - Getting 4 (777.0 B) non-empty blocks including 4 (777.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
21:17:55.754 INFO  ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
21:17:55.755 INFO  ShuffleBlockFetcherIterator - Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
21:17:55.755 INFO  ShuffleBlockFetcherIterator - Started 0 remote fetches in 7 ms
21:17:55.756 INFO  Executor - Finished task 0.0 in stage 26.0 (TID 56). 1758 bytes result sent to driver
21:17:55.756 INFO  Executor - Finished task 2.0 in stage 26.0 (TID 57). 1758 bytes result sent to driver
21:17:55.757 INFO  TaskSetManager - Finished task 0.0 in stage 26.0 (TID 56) in 15 ms on localhost (executor driver) (1/4)
21:17:55.758 INFO  Executor - Finished task 3.0 in stage 26.0 (TID 58). 1758 bytes result sent to driver
21:17:55.760 INFO  TaskSetManager - Finished task 2.0 in stage 26.0 (TID 57) in 17 ms on localhost (executor driver) (2/4)
21:17:55.761 INFO  TaskSetManager - Finished task 3.0 in stage 26.0 (TID 58) in 18 ms on localhost (executor driver) (3/4)
21:17:55.762 INFO  BlockManagerInfo - Removed broadcast_4_piece0 on localhost:40577 in memory (size: 4.5 KiB, free: 1920.0 MiB)
21:17:55.771 INFO  Executor - Finished task 1.0 in stage 26.0 (TID 55). 1871 bytes result sent to driver
21:17:55.773 INFO  TaskSetManager - Finished task 1.0 in stage 26.0 (TID 55) in 31 ms on localhost (executor driver) (4/4)
21:17:55.773 INFO  TaskSchedulerImpl - Removed TaskSet 26.0, whose tasks have all completed, from pool 
21:17:55.774 INFO  DAGScheduler - ResultStage 26 (collectAsMap at SparkSharderUnitTest.java:249) finished in 0.039 s
21:17:55.774 INFO  DAGScheduler - Job 18 is finished. Cancelling potential speculative or zombie tasks for this job
21:17:55.774 INFO  TaskSchedulerImpl - Killing all running tasks in stage 26: Stage finished
21:17:55.774 INFO  DAGScheduler - Job 18 finished: collectAsMap at SparkSharderUnitTest.java:249, took 0.104857 s
21:17:55.777 INFO  MemoryStore - Block broadcast_31 stored as values in memory (estimated size 616.0 B, free 1919.9 MiB)
21:17:55.782 INFO  MemoryStore - Block broadcast_31_piece0 stored as bytes in memory (estimated size 320.0 B, free 1919.9 MiB)
21:17:55.783 INFO  BlockManagerInfo - Added broadcast_31_piece0 in memory on localhost:40577 (size: 320.0 B, free: 1920.0 MiB)
21:17:55.788 INFO  SparkContext - Created broadcast 31 from broadcast at SparkSharder.java:100
21:17:55.797 INFO  BlockManagerInfo - Removed broadcast_3_piece0 on localhost:40577 in memory (size: 4.3 KiB, free: 1920.0 MiB)
21:17:55.809 INFO  BlockManagerInfo - Removed broadcast_19_piece0 on localhost:40577 in memory (size: 2.4 KiB, free: 1920.0 MiB)
21:17:55.822 INFO  SparkContext - Starting job: collect at SparkSharderUnitTest.java:254
21:17:55.824 INFO  BlockManagerInfo - Removed broadcast_9_piece0 on localhost:40577 in memory (size: 4.7 KiB, free: 1920.0 MiB)
21:17:55.826 INFO  DAGScheduler - Registering RDD 59 (flatMapToPair at SparkSharder.java:101) as input to shuffle 6
21:17:55.827 INFO  DAGScheduler - Got job 19 (collect at SparkSharderUnitTest.java:254) with 4 output partitions
21:17:55.827 INFO  DAGScheduler - Final stage: ResultStage 28 (collect at SparkSharderUnitTest.java:254)
21:17:55.827 INFO  DAGScheduler - Parents of final stage: List(ShuffleMapStage 27)
21:17:55.828 INFO  DAGScheduler - Missing parents: List(ShuffleMapStage 27)
21:17:55.829 INFO  DAGScheduler - Submitting ShuffleMapStage 27 (MapPartitionsRDD[59] at flatMapToPair at SparkSharder.java:101), which has no missing parents
21:17:55.834 INFO  BlockManagerInfo - Removed broadcast_7_piece0 on localhost:40577 in memory (size: 326.0 B, free: 1920.0 MiB)
21:17:55.835 INFO  MemoryStore - Block broadcast_32 stored as values in memory (estimated size 7.2 KiB, free 1919.9 MiB)
21:17:55.836 INFO  MemoryStore - Block broadcast_32_piece0 stored as bytes in memory (estimated size 3.8 KiB, free 1919.9 MiB)
21:17:55.836 INFO  BlockManagerInfo - Added broadcast_32_piece0 in memory on localhost:40577 (size: 3.8 KiB, free: 1920.0 MiB)
21:17:55.839 INFO  SparkContext - Created broadcast 32 from broadcast at DAGScheduler.scala:1580
21:17:55.839 INFO  DAGScheduler - Submitting 4 missing tasks from ShuffleMapStage 27 (MapPartitionsRDD[59] at flatMapToPair at SparkSharder.java:101) (first 15 tasks are for partitions Vector(0, 1, 2, 3))
21:17:55.839 INFO  TaskSchedulerImpl - Adding task set 27.0 with 4 tasks resource profile 0
21:17:55.841 INFO  BlockManagerInfo - Removed broadcast_0_piece0 on localhost:40577 in memory (size: 2.4 KiB, free: 1920.0 MiB)
21:17:55.845 INFO  TaskSetManager - Starting task 0.0 in stage 27.0 (TID 59) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7681 bytes) 
21:17:55.847 INFO  TaskSetManager - Starting task 1.0 in stage 27.0 (TID 60) (localhost, executor driver, partition 1, PROCESS_LOCAL, 7681 bytes) 
21:17:55.849 INFO  TaskSetManager - Starting task 2.0 in stage 27.0 (TID 61) (localhost, executor driver, partition 2, PROCESS_LOCAL, 7681 bytes) 
21:17:55.851 INFO  TaskSetManager - Starting task 3.0 in stage 27.0 (TID 62) (localhost, executor driver, partition 3, PROCESS_LOCAL, 7681 bytes) 
21:17:55.851 INFO  Executor - Running task 0.0 in stage 27.0 (TID 59)
21:17:55.853 INFO  Executor - Running task 3.0 in stage 27.0 (TID 62)
21:17:55.857 INFO  Executor - Running task 1.0 in stage 27.0 (TID 60)
21:17:55.858 INFO  Executor - Running task 2.0 in stage 27.0 (TID 61)
21:17:55.874 INFO  Executor - Finished task 3.0 in stage 27.0 (TID 62). 1065 bytes result sent to driver
21:17:55.880 INFO  Executor - Finished task 0.0 in stage 27.0 (TID 59). 1065 bytes result sent to driver
21:17:55.881 INFO  Executor - Finished task 2.0 in stage 27.0 (TID 61). 1065 bytes result sent to driver
21:17:55.885 INFO  TaskSetManager - Finished task 3.0 in stage 27.0 (TID 62) in 34 ms on localhost (executor driver) (1/4)
21:17:55.885 INFO  TaskSetManager - Finished task 2.0 in stage 27.0 (TID 61) in 36 ms on localhost (executor driver) (2/4)
21:17:55.886 INFO  TaskSetManager - Finished task 0.0 in stage 27.0 (TID 59) in 40 ms on localhost (executor driver) (3/4)
21:17:55.890 INFO  Executor - Finished task 1.0 in stage 27.0 (TID 60). 1065 bytes result sent to driver
21:17:55.891 INFO  TaskSetManager - Finished task 1.0 in stage 27.0 (TID 60) in 44 ms on localhost (executor driver) (4/4)
21:17:55.891 INFO  TaskSchedulerImpl - Removed TaskSet 27.0, whose tasks have all completed, from pool 
21:17:55.891 INFO  DAGScheduler - ShuffleMapStage 27 (flatMapToPair at SparkSharder.java:101) finished in 0.057 s
21:17:55.891 INFO  DAGScheduler - looking for newly runnable stages
21:17:55.891 INFO  DAGScheduler - running: HashSet()
21:17:55.891 INFO  DAGScheduler - waiting: HashSet(ResultStage 28)
21:17:55.892 INFO  DAGScheduler - failed: HashSet()
21:17:55.892 INFO  DAGScheduler - Submitting ResultStage 28 (MapPartitionsRDD[63] at flatMapToPair at SparkSharderUnitTest.java:254), which has no missing parents
21:17:55.896 INFO  MemoryStore - Block broadcast_33 stored as values in memory (estimated size 9.4 KiB, free 1919.9 MiB)
21:17:55.897 INFO  MemoryStore - Block broadcast_33_piece0 stored as bytes in memory (estimated size 4.8 KiB, free 1919.9 MiB)
21:17:55.898 INFO  BlockManagerInfo - Added broadcast_33_piece0 in memory on localhost:40577 (size: 4.8 KiB, free: 1920.0 MiB)
21:17:55.898 INFO  SparkContext - Created broadcast 33 from broadcast at DAGScheduler.scala:1580
21:17:55.899 INFO  DAGScheduler - Submitting 4 missing tasks from ResultStage 28 (MapPartitionsRDD[63] at flatMapToPair at SparkSharderUnitTest.java:254) (first 15 tasks are for partitions Vector(0, 1, 2, 3))
21:17:55.899 INFO  TaskSchedulerImpl - Adding task set 28.0 with 4 tasks resource profile 0
21:17:55.900 INFO  TaskSetManager - Starting task 1.0 in stage 28.0 (TID 63) (localhost, executor driver, partition 1, NODE_LOCAL, 7513 bytes) 
21:17:55.901 INFO  TaskSetManager - Starting task 0.0 in stage 28.0 (TID 64) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7513 bytes) 
21:17:55.902 INFO  TaskSetManager - Starting task 2.0 in stage 28.0 (TID 65) (localhost, executor driver, partition 2, PROCESS_LOCAL, 7513 bytes) 
21:17:55.902 INFO  TaskSetManager - Starting task 3.0 in stage 28.0 (TID 66) (localhost, executor driver, partition 3, PROCESS_LOCAL, 7513 bytes) 
21:17:55.903 INFO  Executor - Running task 1.0 in stage 28.0 (TID 63)
21:17:55.908 INFO  Executor - Running task 0.0 in stage 28.0 (TID 64)
21:17:55.911 INFO  Executor - Running task 3.0 in stage 28.0 (TID 66)
21:17:55.913 INFO  Executor - Running task 2.0 in stage 28.0 (TID 65)
21:17:55.913 INFO  ShuffleBlockFetcherIterator - Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
21:17:55.913 INFO  ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
21:17:55.915 INFO  ShuffleBlockFetcherIterator - Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
21:17:55.915 INFO  ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
21:17:55.915 INFO  ShuffleBlockFetcherIterator - Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
21:17:55.915 INFO  ShuffleBlockFetcherIterator - Started 0 remote fetches in 4 ms
21:17:55.916 INFO  ShuffleBlockFetcherIterator - Getting 4 (777.0 B) non-empty blocks including 4 (777.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
21:17:55.916 INFO  ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
21:17:55.916 INFO  Executor - Finished task 0.0 in stage 28.0 (TID 64). 1758 bytes result sent to driver
21:17:55.917 INFO  Executor - Finished task 3.0 in stage 28.0 (TID 66). 1758 bytes result sent to driver
21:17:55.918 INFO  Executor - Finished task 2.0 in stage 28.0 (TID 65). 1758 bytes result sent to driver
21:17:55.919 INFO  TaskSetManager - Finished task 0.0 in stage 28.0 (TID 64) in 18 ms on localhost (executor driver) (1/4)
21:17:55.919 INFO  TaskSetManager - Finished task 3.0 in stage 28.0 (TID 66) in 17 ms on localhost (executor driver) (2/4)
21:17:55.919 INFO  TaskSetManager - Finished task 2.0 in stage 28.0 (TID 65) in 18 ms on localhost (executor driver) (3/4)
21:17:55.926 INFO  Executor - Finished task 1.0 in stage 28.0 (TID 63). 1871 bytes result sent to driver
21:17:55.927 INFO  TaskSetManager - Finished task 1.0 in stage 28.0 (TID 63) in 27 ms on localhost (executor driver) (4/4)
21:17:55.927 INFO  TaskSchedulerImpl - Removed TaskSet 28.0, whose tasks have all completed, from pool 
21:17:55.928 INFO  DAGScheduler - ResultStage 28 (collect at SparkSharderUnitTest.java:254) finished in 0.035 s
21:17:55.931 INFO  DAGScheduler - Job 19 is finished. Cancelling potential speculative or zombie tasks for this job
21:17:55.931 INFO  TaskSchedulerImpl - Killing all running tasks in stage 28: Stage finished
21:17:55.931 INFO  DAGScheduler - Job 19 finished: collect at SparkSharderUnitTest.java:254, took 0.108931 s