19:01:58.923 INFO SparkContext - Starting job: collect at SparkSharder.java:388
19:01:58.935 INFO DAGScheduler - Got job 0 (collect at SparkSharder.java:388) with 1 output partitions
19:01:58.935 INFO DAGScheduler - Final stage: ResultStage 0 (collect at SparkSharder.java:388)
19:01:58.936 INFO DAGScheduler - Parents of final stage: List()
19:01:58.937 INFO DAGScheduler - Missing parents: List()
19:01:58.939 INFO DAGScheduler - Submitting ResultStage 0 (MapPartitionsRDD[3] at mapPartitions at SparkSharder.java:386), which has no missing parents
19:01:58.979 INFO MemoryStore - Block broadcast_0 stored as values in memory (estimated size 4.5 KiB, free 1920.0 MiB)
19:01:59.002 INFO MemoryStore - Block broadcast_0_piece0 stored as bytes in memory (estimated size 2.4 KiB, free 1920.0 MiB)
19:01:59.004 INFO BlockManagerInfo - Added broadcast_0_piece0 in memory on localhost:45255 (size: 2.4 KiB, free: 1920.0 MiB)
19:01:59.007 INFO SparkContext - Created broadcast 0 from broadcast at DAGScheduler.scala:1580
19:01:59.020 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 0 (MapPartitionsRDD[3] at mapPartitions at SparkSharder.java:386) (first 15 tasks are for partitions Vector(0))
19:01:59.020 INFO TaskSchedulerImpl - Adding task set 0.0 with 1 tasks resource profile 0
19:01:59.052 INFO TaskSetManager - Starting task 0.0 in stage 0.0 (TID 0) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7718 bytes)
19:01:59.063 INFO Executor - Running task 0.0 in stage 0.0 (TID 0)
19:01:59.133 INFO Executor - Finished task 0.0 in stage 0.0 (TID 0). 1116 bytes result sent to driver
19:01:59.139 INFO TaskSetManager - Finished task 0.0 in stage 0.0 (TID 0) in 95 ms on localhost (executor driver) (1/1)
19:01:59.140 INFO TaskSchedulerImpl - Removed TaskSet 0.0, whose tasks have all completed, from pool
19:01:59.143 INFO DAGScheduler - ResultStage 0 (collect at SparkSharder.java:388) finished in 0.190 s
19:01:59.145 INFO DAGScheduler - Job 0 is finished. Cancelling potential speculative or zombie tasks for this job
19:01:59.145 INFO TaskSchedulerImpl - Killing all running tasks in stage 0: Stage finished
19:01:59.147 INFO DAGScheduler - Job 0 finished: collect at SparkSharder.java:388, took 0.223617 s
19:01:59.153 INFO MemoryStore - Block broadcast_1 stored as values in memory (estimated size 248.0 B, free 1920.0 MiB)
19:01:59.157 INFO MemoryStore - Block broadcast_1_piece0 stored as bytes in memory (estimated size 143.0 B, free 1920.0 MiB)
19:01:59.158 INFO BlockManagerInfo - Added broadcast_1_piece0 in memory on localhost:45255 (size: 143.0 B, free: 1920.0 MiB)
19:01:59.158 INFO SparkContext - Created broadcast 1 from broadcast at SparkSharder.java:214
19:01:59.161 INFO MemoryStore - Block broadcast_2 stored as values in memory (estimated size 608.0 B, free 1920.0 MiB)
19:01:59.165 INFO MemoryStore - Block broadcast_2_piece0 stored as bytes in memory (estimated size 349.0 B, free 1920.0 MiB)
19:01:59.165 INFO BlockManagerInfo - Added broadcast_2_piece0 in memory on localhost:45255 (size: 349.0 B, free: 1920.0 MiB)
19:01:59.166 INFO SparkContext - Created broadcast 2 from broadcast at SparkSharder.java:221
19:01:59.226 INFO SparkContext - Starting job: collectAsMap at SparkSharder.java:258
19:01:59.239 INFO DAGScheduler - Registering RDD 5 (mapToPair at SparkSharder.java:247) as input to shuffle 1
19:01:59.244 INFO DAGScheduler - Registering RDD 8 (mapToPair at SparkSharder.java:255) as input to shuffle 0
19:01:59.244 INFO DAGScheduler - Got job 1 (collectAsMap at SparkSharder.java:258) with 1 output partitions
19:01:59.244 INFO DAGScheduler - Final stage: ResultStage 3 (collectAsMap at SparkSharder.java:258)
19:01:59.245 INFO DAGScheduler - Parents of final stage: List(ShuffleMapStage 2)
19:01:59.245 INFO DAGScheduler - Missing parents: List(ShuffleMapStage 2)
19:01:59.248 INFO DAGScheduler - Submitting ShuffleMapStage 1 (MapPartitionsRDD[5] at mapToPair at SparkSharder.java:247), which has no missing parents
19:01:59.258 INFO MemoryStore - Block broadcast_3 stored as values in memory (estimated size 7.9 KiB, free 1920.0 MiB)
19:01:59.259 INFO MemoryStore - Block broadcast_3_piece0 stored as bytes in memory (estimated size 4.3 KiB, free 1920.0 MiB)
19:01:59.259 INFO BlockManagerInfo - Added broadcast_3_piece0 in memory on localhost:45255 (size: 4.3 KiB, free: 1920.0 MiB)
19:01:59.260 INFO SparkContext - Created broadcast 3 from broadcast at DAGScheduler.scala:1580
19:01:59.261 INFO DAGScheduler - Submitting 4 missing tasks from ShuffleMapStage 1 (MapPartitionsRDD[5] at mapToPair at SparkSharder.java:247) (first 15 tasks are for partitions Vector(0, 1, 2, 3))
19:01:59.261 INFO TaskSchedulerImpl - Adding task set 1.0 with 4 tasks resource profile 0
19:01:59.263 INFO TaskSetManager - Starting task 0.0 in stage 1.0 (TID 1) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7569 bytes)
19:01:59.264 INFO TaskSetManager - Starting task 1.0 in stage 1.0 (TID 2) (localhost, executor driver, partition 1, PROCESS_LOCAL, 7636 bytes)
19:01:59.264 INFO TaskSetManager - Starting task 2.0 in stage 1.0 (TID 3) (localhost, executor driver, partition 2, PROCESS_LOCAL, 7636 bytes)
19:01:59.265 INFO TaskSetManager - Starting task 3.0 in stage 1.0 (TID 4) (localhost, executor driver, partition 3, PROCESS_LOCAL, 7636 bytes)
19:01:59.265 INFO Executor - Running task 0.0 in stage 1.0 (TID 1)
19:01:59.265 INFO Executor - Running task 1.0 in stage 1.0 (TID 2)
19:01:59.267 INFO Executor - Running task 2.0 in stage 1.0 (TID 3)
19:01:59.268 INFO Executor - Running task 3.0 in stage 1.0 (TID 4)
19:01:59.302 INFO Executor - Finished task 0.0 in stage 1.0 (TID 1). 933 bytes result sent to driver
19:01:59.316 INFO TaskSetManager - Finished task 0.0 in stage 1.0 (TID 1) in 54 ms on localhost (executor driver) (1/4)
19:01:59.317 INFO Executor - Finished task 1.0 in stage 1.0 (TID 2). 1062 bytes result sent to driver
19:01:59.319 INFO TaskSetManager - Finished task 1.0 in stage 1.0 (TID 2) in 55 ms on localhost (executor driver) (2/4)
19:01:59.320 INFO Executor - Finished task 2.0 in stage 1.0 (TID 3). 1105 bytes result sent to driver
19:01:59.321 INFO Executor - Finished task 3.0 in stage 1.0 (TID 4). 1062 bytes result sent to driver
19:01:59.322 INFO TaskSetManager - Finished task 3.0 in stage 1.0 (TID 4) in 58 ms on localhost (executor driver) (3/4)
19:01:59.322 INFO TaskSetManager - Finished task 2.0 in stage 1.0 (TID 3) in 58 ms on localhost (executor driver) (4/4)
19:01:59.322 INFO TaskSchedulerImpl - Removed TaskSet 1.0, whose tasks have all completed, from pool
19:01:59.323 INFO DAGScheduler - ShuffleMapStage 1 (mapToPair at SparkSharder.java:247) finished in 0.073 s
19:01:59.324 INFO DAGScheduler - looking for newly runnable stages
19:01:59.325 INFO DAGScheduler - running: HashSet()
19:01:59.325 INFO DAGScheduler - waiting: HashSet(ShuffleMapStage 2, ResultStage 3)
19:01:59.325 INFO DAGScheduler - failed: HashSet()
19:01:59.328 INFO DAGScheduler - Submitting ShuffleMapStage 2 (MapPartitionsRDD[8] at mapToPair at SparkSharder.java:255), which has no missing parents
19:01:59.336 INFO MemoryStore - Block broadcast_4 stored as values in memory (estimated size 8.2 KiB, free 1920.0 MiB)
19:01:59.339 INFO MemoryStore - Block broadcast_4_piece0 stored as bytes in memory (estimated size 4.5 KiB, free 1920.0 MiB)
19:01:59.339 INFO BlockManagerInfo - Added broadcast_4_piece0 in memory on localhost:45255 (size: 4.5 KiB, free: 1920.0 MiB)
19:01:59.340 INFO SparkContext - Created broadcast 4 from broadcast at DAGScheduler.scala:1580
19:01:59.340 INFO DAGScheduler - Submitting 1 missing tasks from ShuffleMapStage 2 (MapPartitionsRDD[8] at mapToPair at SparkSharder.java:255) (first 15 tasks are for partitions Vector(0))
19:01:59.340 INFO TaskSchedulerImpl - Adding task set 2.0 with 1 tasks resource profile 0
19:01:59.344 INFO TaskSetManager - Starting task 0.0 in stage 2.0 (TID 5) (localhost, executor driver, partition 0, NODE_LOCAL, 7502 bytes)
19:01:59.345 INFO Executor - Running task 0.0 in stage 2.0 (TID 5)
19:01:59.383 INFO ShuffleBlockFetcherIterator - Getting 3 (513.0 B) non-empty blocks including 3 (513.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
19:01:59.384 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 8 ms
19:01:59.414 INFO MemoryStore - Block rdd_7_0 stored as values in memory (estimated size 392.0 B, free 1920.0 MiB)
19:01:59.415 INFO BlockManagerInfo - Added rdd_7_0 in memory on localhost:45255 (size: 392.0 B, free: 1920.0 MiB)
19:01:59.425 INFO Executor - Finished task 0.0 in stage 2.0 (TID 5). 1922 bytes result sent to driver
19:01:59.426 INFO TaskSetManager - Finished task 0.0 in stage 2.0 (TID 5) in 83 ms on localhost (executor driver) (1/1)
19:01:59.427 INFO TaskSchedulerImpl - Removed TaskSet 2.0, whose tasks have all completed, from pool
19:01:59.429 INFO DAGScheduler - ShuffleMapStage 2 (mapToPair at SparkSharder.java:255) finished in 0.096 s
19:01:59.429 INFO DAGScheduler - looking for newly runnable stages
19:01:59.429 INFO DAGScheduler - running: HashSet()
19:01:59.429 INFO DAGScheduler - waiting: HashSet(ResultStage 3)
19:01:59.429 INFO DAGScheduler - failed: HashSet()
19:01:59.429 INFO DAGScheduler - Submitting ResultStage 3 (ShuffledRDD[9] at reduceByKey at SparkSharder.java:257), which has no missing parents
19:01:59.432 INFO MemoryStore - Block broadcast_5 stored as values in memory (estimated size 5.8 KiB, free 1920.0 MiB)
19:01:59.435 INFO MemoryStore - Block broadcast_5_piece0 stored as bytes in memory (estimated size 3.2 KiB, free 1920.0 MiB)
19:01:59.436 INFO BlockManagerInfo - Added broadcast_5_piece0 in memory on localhost:45255 (size: 3.2 KiB, free: 1920.0 MiB)
19:01:59.436 INFO SparkContext - Created broadcast 5 from broadcast at DAGScheduler.scala:1580
19:01:59.437 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 3 (ShuffledRDD[9] at reduceByKey at SparkSharder.java:257) (first 15 tasks are for partitions Vector(0))
19:01:59.437 INFO TaskSchedulerImpl - Adding task set 3.0 with 1 tasks resource profile 0
19:01:59.438 INFO TaskSetManager - Starting task 0.0 in stage 3.0 (TID 6) (localhost, executor driver, partition 0, NODE_LOCAL, 7513 bytes)
19:01:59.439 INFO Executor - Running task 0.0 in stage 3.0 (TID 6)
19:01:59.444 INFO ShuffleBlockFetcherIterator - Getting 1 (54.0 B) non-empty blocks including 1 (54.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
19:01:59.444 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
19:01:59.457 INFO Executor - Finished task 0.0 in stage 3.0 (TID 6). 1770 bytes result sent to driver
19:01:59.459 INFO TaskSetManager - Finished task 0.0 in stage 3.0 (TID 6) in 21 ms on localhost (executor driver) (1/1)
19:01:59.459 INFO TaskSchedulerImpl - Removed TaskSet 3.0, whose tasks have all completed, from pool
19:01:59.460 INFO DAGScheduler - ResultStage 3 (collectAsMap at SparkSharder.java:258) finished in 0.030 s
19:01:59.460 INFO DAGScheduler - Job 1 is finished. Cancelling potential speculative or zombie tasks for this job
19:01:59.460 INFO TaskSchedulerImpl - Killing all running tasks in stage 3: Stage finished
19:01:59.460 INFO DAGScheduler - Job 1 finished: collectAsMap at SparkSharder.java:258, took 0.234059 s
19:01:59.505 INFO SparkContext - Starting job: collectAsMap at SparkSharderUnitTest.java:305
19:01:59.506 INFO DAGScheduler - Got job 2 (collectAsMap at SparkSharderUnitTest.java:305) with 1 output partitions
19:01:59.506 INFO DAGScheduler - Final stage: ResultStage 5 (collectAsMap at SparkSharderUnitTest.java:305)
19:01:59.506 INFO DAGScheduler - Parents of final stage: List(ShuffleMapStage 4)
19:01:59.507 INFO DAGScheduler - Missing parents: List()
19:01:59.509 INFO DAGScheduler - Submitting ResultStage 5 (MapPartitionsRDD[13] at flatMapToPair at SparkSharderUnitTest.java:304), which has no missing parents
19:01:59.513 INFO MemoryStore - Block broadcast_6 stored as values in memory (estimated size 10.2 KiB, free 1919.9 MiB)
19:01:59.515 INFO MemoryStore - Block broadcast_6_piece0 stored as bytes in memory (estimated size 5.0 KiB, free 1919.9 MiB)
19:01:59.516 INFO BlockManagerInfo - Added broadcast_6_piece0 in memory on localhost:45255 (size: 5.0 KiB, free: 1920.0 MiB)
19:01:59.516 INFO SparkContext - Created broadcast 6 from broadcast at DAGScheduler.scala:1580
19:01:59.517 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 5 (MapPartitionsRDD[13] at flatMapToPair at SparkSharderUnitTest.java:304) (first 15 tasks are for partitions Vector(0))
19:01:59.517 INFO TaskSchedulerImpl - Adding task set 5.0 with 1 tasks resource profile 0
19:01:59.523 INFO TaskSetManager - Starting task 0.0 in stage 5.0 (TID 7) (localhost, executor driver, partition 0, PROCESS_LOCAL, 8559 bytes)
19:01:59.523 INFO Executor - Running task 0.0 in stage 5.0 (TID 7)
19:01:59.530 INFO BlockManager - Found block rdd_7_0 locally
19:01:59.533 INFO Executor - Finished task 0.0 in stage 5.0 (TID 7). 1097 bytes result sent to driver
19:01:59.534 INFO TaskSetManager - Finished task 0.0 in stage 5.0 (TID 7) in 16 ms on localhost (executor driver) (1/1)
19:01:59.534 INFO TaskSchedulerImpl - Removed TaskSet 5.0, whose tasks have all completed, from pool
19:01:59.535 INFO DAGScheduler - ResultStage 5 (collectAsMap at SparkSharderUnitTest.java:305) finished in 0.025 s
19:01:59.535 INFO DAGScheduler - Job 2 is finished. Cancelling potential speculative or zombie tasks for this job
19:01:59.535 INFO TaskSchedulerImpl - Killing all running tasks in stage 5: Stage finished
19:01:59.536 INFO DAGScheduler - Job 2 finished: collectAsMap at SparkSharderUnitTest.java:305, took 0.031526 s
19:01:59.539 INFO MemoryStore - Block broadcast_7 stored as values in memory (estimated size 720.0 B, free 1919.9 MiB)
19:01:59.542 INFO MemoryStore - Block broadcast_7_piece0 stored as bytes in memory (estimated size 326.0 B, free 1919.9 MiB)
19:01:59.543 INFO BlockManagerInfo - Added broadcast_7_piece0 in memory on localhost:45255 (size: 326.0 B, free: 1920.0 MiB)
19:01:59.543 INFO SparkContext - Created broadcast 7 from broadcast at SparkSharder.java:100
19:01:59.583 INFO SparkContext - Starting job: collectAsMap at SparkSharderUnitTest.java:310
19:01:59.584 INFO DAGScheduler - Registering RDD 14 (flatMapToPair at SparkSharder.java:101) as input to shuffle 2
19:01:59.584 INFO DAGScheduler - Got job 3 (collectAsMap at SparkSharderUnitTest.java:310) with 1 output partitions
19:01:59.584 INFO DAGScheduler - Final stage: ResultStage 7 (collectAsMap at SparkSharderUnitTest.java:310)
19:01:59.584 INFO DAGScheduler - Parents of final stage: List(ShuffleMapStage 6)
19:01:59.585 INFO DAGScheduler - Missing parents: List(ShuffleMapStage 6)
19:01:59.586 INFO DAGScheduler - Submitting ShuffleMapStage 6 (MapPartitionsRDD[14] at flatMapToPair at SparkSharder.java:101), which has no missing parents
19:01:59.589 INFO MemoryStore - Block broadcast_8 stored as values in memory (estimated size 7.2 KiB, free 1919.9 MiB)
19:01:59.592 INFO MemoryStore - Block broadcast_8_piece0 stored as bytes in memory (estimated size 3.8 KiB, free 1919.9 MiB)
19:01:59.592 INFO BlockManagerInfo - Added broadcast_8_piece0 in memory on localhost:45255 (size: 3.8 KiB, free: 1920.0 MiB)
19:01:59.593 INFO SparkContext - Created broadcast 8 from broadcast at DAGScheduler.scala:1580
19:01:59.593 INFO DAGScheduler - Submitting 1 missing tasks from ShuffleMapStage 6 (MapPartitionsRDD[14] at flatMapToPair at SparkSharder.java:101) (first 15 tasks are for partitions Vector(0))
19:01:59.593 INFO TaskSchedulerImpl - Adding task set 6.0 with 1 tasks resource profile 0
19:01:59.595 INFO TaskSetManager - Starting task 0.0 in stage 6.0 (TID 8) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7707 bytes)
19:01:59.596 INFO Executor - Running task 0.0 in stage 6.0 (TID 8)
19:01:59.610 INFO Executor - Finished task 0.0 in stage 6.0 (TID 8). 1062 bytes result sent to driver
19:01:59.611 INFO TaskSetManager - Finished task 0.0 in stage 6.0 (TID 8) in 17 ms on localhost (executor driver) (1/1)
19:01:59.611 INFO TaskSchedulerImpl - Removed TaskSet 6.0, whose tasks have all completed, from pool
19:01:59.612 INFO DAGScheduler - ShuffleMapStage 6 (flatMapToPair at SparkSharder.java:101) finished in 0.025 s
19:01:59.612 INFO DAGScheduler - looking for newly runnable stages
19:01:59.612 INFO DAGScheduler - running: HashSet()
19:01:59.612 INFO DAGScheduler - waiting: HashSet(ResultStage 7)
19:01:59.612 INFO DAGScheduler - failed: HashSet()
19:01:59.613 INFO DAGScheduler - Submitting ResultStage 7 (MapPartitionsRDD[18] at flatMapToPair at SparkSharderUnitTest.java:309), which has no missing parents
19:01:59.615 INFO MemoryStore - Block broadcast_9 stored as values in memory (estimated size 9.4 KiB, free 1919.9 MiB)
19:01:59.618 INFO MemoryStore - Block broadcast_9_piece0 stored as bytes in memory (estimated size 4.7 KiB, free 1919.9 MiB)
19:01:59.618 INFO BlockManagerInfo - Added broadcast_9_piece0 in memory on localhost:45255 (size: 4.7 KiB, free: 1920.0 MiB)
19:01:59.619 INFO SparkContext - Created broadcast 9 from broadcast at DAGScheduler.scala:1580
19:01:59.619 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 7 (MapPartitionsRDD[18] at flatMapToPair at SparkSharderUnitTest.java:309) (first 15 tasks are for partitions Vector(0))
19:01:59.619 INFO TaskSchedulerImpl - Adding task set 7.0 with 1 tasks resource profile 0
19:01:59.621 INFO TaskSetManager - Starting task 0.0 in stage 7.0 (TID 9) (localhost, executor driver, partition 0, NODE_LOCAL, 7513 bytes)
19:01:59.621 INFO Executor - Running task 0.0 in stage 7.0 (TID 9)
19:01:59.627 INFO ShuffleBlockFetcherIterator - Getting 1 (228.0 B) non-empty blocks including 1 (228.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
19:01:59.627 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
19:01:59.638 INFO Executor - Finished task 0.0 in stage 7.0 (TID 9). 1871 bytes result sent to driver
19:01:59.640 INFO TaskSetManager - Finished task 0.0 in stage 7.0 (TID 9) in 20 ms on localhost (executor driver) (1/1)
19:01:59.640 INFO TaskSchedulerImpl - Removed TaskSet 7.0, whose tasks have all completed, from pool
19:01:59.641 INFO DAGScheduler - ResultStage 7 (collectAsMap at SparkSharderUnitTest.java:310) finished in 0.028 s
19:01:59.641 INFO DAGScheduler - Job 3 is finished. Cancelling potential speculative or zombie tasks for this job
19:01:59.641 INFO TaskSchedulerImpl - Killing all running tasks in stage 7: Stage finished
19:01:59.642 INFO DAGScheduler - Job 3 finished: collectAsMap at SparkSharderUnitTest.java:310, took 0.058273 s
19:01:59.660 INFO SparkContext - Starting job: collect at SparkSharder.java:388
19:01:59.661 INFO DAGScheduler - Got job 4 (collect at SparkSharder.java:388) with 1 output partitions
19:01:59.661 INFO DAGScheduler - Final stage: ResultStage 8 (collect at SparkSharder.java:388)
19:01:59.661 INFO DAGScheduler - Parents of final stage: List()
19:01:59.661 INFO DAGScheduler - Missing parents: List()
19:01:59.661 INFO DAGScheduler - Submitting ResultStage 8 (MapPartitionsRDD[20] at mapPartitions at SparkSharder.java:386), which has no missing parents
19:01:59.663 INFO MemoryStore - Block broadcast_10 stored as values in memory (estimated size 4.5 KiB, free 1919.9 MiB)
19:01:59.666 INFO MemoryStore - Block broadcast_10_piece0 stored as bytes in memory (estimated size 2.4 KiB, free 1919.9 MiB)
19:01:59.667 INFO BlockManagerInfo - Added broadcast_10_piece0 in memory on localhost:45255 (size: 2.4 KiB, free: 1920.0 MiB)
19:01:59.667 INFO SparkContext - Created broadcast 10 from broadcast at DAGScheduler.scala:1580
19:01:59.668 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 8 (MapPartitionsRDD[20] at mapPartitions at SparkSharder.java:386) (first 15 tasks are for partitions Vector(0))
19:01:59.668 INFO TaskSchedulerImpl - Adding task set 8.0 with 1 tasks resource profile 0
19:01:59.669 INFO TaskSetManager - Starting task 0.0 in stage 8.0 (TID 10) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7730 bytes)
19:01:59.670 INFO Executor - Running task 0.0 in stage 8.0 (TID 10)
19:01:59.675 INFO Executor - Finished task 0.0 in stage 8.0 (TID 10). 1030 bytes result sent to driver
19:01:59.677 INFO TaskSetManager - Finished task 0.0 in stage 8.0 (TID 10) in 8 ms on localhost (executor driver) (1/1)
19:01:59.677 INFO TaskSchedulerImpl - Removed TaskSet 8.0, whose tasks have all completed, from pool
19:01:59.678 INFO DAGScheduler - ResultStage 8 (collect at SparkSharder.java:388) finished in 0.016 s
19:01:59.678 INFO DAGScheduler - Job 4 is finished. Cancelling potential speculative or zombie tasks for this job
19:01:59.678 INFO TaskSchedulerImpl - Killing all running tasks in stage 8: Stage finished
19:01:59.681 INFO DAGScheduler - Job 4 finished: collect at SparkSharder.java:388, took 0.019223 s
19:01:59.695 INFO SparkContext - Starting job: collect at SparkSharder.java:388
19:01:59.696 INFO DAGScheduler - Got job 5 (collect at SparkSharder.java:388) with 2 output partitions
19:01:59.696 INFO DAGScheduler - Final stage: ResultStage 9 (collect at SparkSharder.java:388)
19:01:59.696 INFO DAGScheduler - Parents of final stage: List()
19:01:59.696 INFO DAGScheduler - Missing parents: List()
19:01:59.697 INFO DAGScheduler - Submitting ResultStage 9 (MapPartitionsRDD[22] at mapPartitions at SparkSharder.java:386), which has no missing parents
19:01:59.699 INFO MemoryStore - Block broadcast_11 stored as values in memory (estimated size 4.5 KiB, free 1919.9 MiB)
19:01:59.702 INFO MemoryStore - Block broadcast_11_piece0 stored as bytes in memory (estimated size 2.4 KiB, free 1919.9 MiB)
19:01:59.703 INFO BlockManagerInfo - Added broadcast_11_piece0 in memory on localhost:45255 (size: 2.4 KiB, free: 1920.0 MiB)
19:01:59.703 INFO SparkContext - Created broadcast 11 from broadcast at DAGScheduler.scala:1580
19:01:59.703 INFO DAGScheduler - Submitting 2 missing tasks from ResultStage 9 (MapPartitionsRDD[22] at mapPartitions at SparkSharder.java:386) (first 15 tasks are for partitions Vector(0, 1))
19:01:59.703 INFO TaskSchedulerImpl - Adding task set 9.0 with 2 tasks resource profile 0
19:01:59.705 INFO TaskSetManager - Starting task 0.0 in stage 9.0 (TID 11) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7692 bytes)
19:01:59.705 INFO TaskSetManager - Starting task 1.0 in stage 9.0 (TID 12) (localhost, executor driver, partition 1, PROCESS_LOCAL, 7692 bytes)
19:01:59.706 INFO Executor - Running task 0.0 in stage 9.0 (TID 11)
19:01:59.706 INFO Executor - Running task 1.0 in stage 9.0 (TID 12)
19:01:59.710 INFO Executor - Finished task 0.0 in stage 9.0 (TID 11). 1073 bytes result sent to driver
19:01:59.710 INFO Executor - Finished task 1.0 in stage 9.0 (TID 12). 1073 bytes result sent to driver
19:01:59.711 INFO TaskSetManager - Finished task 0.0 in stage 9.0 (TID 11) in 7 ms on localhost (executor driver) (1/2)
19:01:59.711 INFO TaskSetManager - Finished task 1.0 in stage 9.0 (TID 12) in 6 ms on localhost (executor driver) (2/2)
19:01:59.711 INFO TaskSchedulerImpl - Removed TaskSet 9.0, whose tasks have all completed, from pool
19:01:59.712 INFO DAGScheduler - ResultStage 9 (collect at SparkSharder.java:388) finished in 0.014 s
19:01:59.712 INFO DAGScheduler - Job 5 is finished. Cancelling potential speculative or zombie tasks for this job
19:01:59.712 INFO TaskSchedulerImpl - Killing all running tasks in stage 9: Stage finished
19:01:59.712 INFO DAGScheduler - Job 5 finished: collect at SparkSharder.java:388, took 0.017108 s
19:01:59.728 INFO SparkContext - Starting job: collect at SparkSharder.java:388
19:01:59.729 INFO DAGScheduler - Got job 6 (collect at SparkSharder.java:388) with 3 output partitions
19:01:59.729 INFO DAGScheduler - Final stage: ResultStage 10 (collect at SparkSharder.java:388)
19:01:59.729 INFO DAGScheduler - Parents of final stage: List()
19:01:59.729 INFO DAGScheduler - Missing parents: List()
19:01:59.730 INFO DAGScheduler - Submitting ResultStage 10 (MapPartitionsRDD[24] at mapPartitions at SparkSharder.java:386), which has no missing parents
19:01:59.732 INFO MemoryStore - Block broadcast_12 stored as values in memory (estimated size 4.5 KiB, free 1919.9 MiB)
19:01:59.735 INFO MemoryStore - Block broadcast_12_piece0 stored as bytes in memory (estimated size 2.4 KiB, free 1919.9 MiB)
19:01:59.735 INFO BlockManagerInfo - Added broadcast_12_piece0 in memory on localhost:45255 (size: 2.4 KiB, free: 1920.0 MiB)
19:01:59.736 INFO SparkContext - Created broadcast 12 from broadcast at DAGScheduler.scala:1580
19:01:59.736 INFO DAGScheduler - Submitting 3 missing tasks from ResultStage 10 (MapPartitionsRDD[24] at mapPartitions at SparkSharder.java:386) (first 15 tasks are for partitions Vector(0, 1, 2))
19:01:59.736 INFO TaskSchedulerImpl - Adding task set 10.0 with 3 tasks resource profile 0
19:01:59.738 INFO TaskSetManager - Starting task 0.0 in stage 10.0 (TID 13) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7680 bytes)
19:01:59.738 INFO TaskSetManager - Starting task 1.0 in stage 10.0 (TID 14) (localhost, executor driver, partition 1, PROCESS_LOCAL, 7682 bytes)
19:01:59.739 INFO TaskSetManager - Starting task 2.0 in stage 10.0 (TID 15) (localhost, executor driver, partition 2, PROCESS_LOCAL, 7680 bytes)
19:01:59.739 INFO Executor - Running task 1.0 in stage 10.0 (TID 14)
19:01:59.740 INFO Executor - Running task 0.0 in stage 10.0 (TID 13)
19:01:59.740 INFO Executor - Running task 2.0 in stage 10.0 (TID 15)
19:01:59.746 INFO Executor - Finished task 0.0 in stage 10.0 (TID 13). 1030 bytes result sent to driver
19:01:59.747 INFO Executor - Finished task 1.0 in stage 10.0 (TID 14). 1030 bytes result sent to driver
19:01:59.748 INFO Executor - Finished task 2.0 in stage 10.0 (TID 15). 1030 bytes result sent to driver
19:01:59.749 INFO TaskSetManager - Finished task 2.0 in stage 10.0 (TID 15) in 11 ms on localhost (executor driver) (1/3)
19:01:59.750 INFO TaskSetManager - Finished task 0.0 in stage 10.0 (TID 13) in 13 ms on localhost (executor driver) (2/3)
19:01:59.751 INFO TaskSetManager - Finished task 1.0 in stage 10.0 (TID 14) in 13 ms on localhost (executor driver) (3/3)
19:01:59.751 INFO TaskSchedulerImpl - Removed TaskSet 10.0, whose tasks have all completed, from pool
19:01:59.751 INFO DAGScheduler - ResultStage 10 (collect at SparkSharder.java:388) finished in 0.020 s
19:01:59.751 INFO DAGScheduler - Job 6 is finished. Cancelling potential speculative or zombie tasks for this job
19:01:59.751 INFO TaskSchedulerImpl - Killing all running tasks in stage 10: Stage finished
19:01:59.751 INFO DAGScheduler - Job 6 finished: collect at SparkSharder.java:388, took 0.022745 s
19:01:59.767 INFO SparkContext - Starting job: collect at SparkSharder.java:388
19:01:59.767 INFO DAGScheduler - Got job 7 (collect at SparkSharder.java:388) with 1 output partitions
19:01:59.767 INFO DAGScheduler - Final stage: ResultStage 11 (collect at SparkSharder.java:388)
19:01:59.767 INFO DAGScheduler - Parents of final stage: List()
19:01:59.767 INFO DAGScheduler - Missing parents: List()
19:01:59.768 INFO DAGScheduler - Submitting ResultStage 11 (MapPartitionsRDD[26] at mapPartitions at SparkSharder.java:386), which has no missing parents
19:01:59.769 INFO MemoryStore - Block broadcast_13 stored as values in memory (estimated size 4.5 KiB, free 1919.9 MiB)
19:01:59.772 INFO MemoryStore - Block broadcast_13_piece0 stored as bytes in memory (estimated size 2.4 KiB, free 1919.9 MiB)
19:01:59.773 INFO BlockManagerInfo - Added broadcast_13_piece0 in memory on localhost:45255 (size: 2.4 KiB, free: 1920.0 MiB)
19:01:59.773 INFO SparkContext - Created broadcast 13 from broadcast at DAGScheduler.scala:1580
19:01:59.774 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 11 (MapPartitionsRDD[26] at mapPartitions at SparkSharder.java:386) (first 15 tasks are for partitions Vector(0))
19:01:59.774 INFO TaskSchedulerImpl - Adding task set 11.0 with 1 tasks resource profile 0
19:01:59.775 INFO TaskSetManager - Starting task 0.0 in stage 11.0 (TID 16) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7730 bytes)
19:01:59.775 INFO Executor - Running task 0.0 in stage 11.0 (TID 16)
19:01:59.779 INFO Executor - Finished task 0.0 in stage 11.0 (TID 16). 1030 bytes result sent to driver
19:01:59.780 INFO TaskSetManager - Finished task 0.0 in stage 11.0 (TID 16) in 6 ms on localhost (executor driver) (1/1)
19:01:59.780 INFO DAGScheduler - ResultStage 11 (collect at SparkSharder.java:388) finished in 0.012 s
19:01:59.780 INFO DAGScheduler - Job 7 is finished. Cancelling potential speculative or zombie tasks for this job
19:01:59.780 INFO TaskSchedulerImpl - Removed TaskSet 11.0, whose tasks have all completed, from pool
19:01:59.781 INFO TaskSchedulerImpl - Killing all running tasks in stage 11: Stage finished
19:01:59.781 INFO DAGScheduler - Job 7 finished: collect at SparkSharder.java:388, took 0.013816 s
19:01:59.798 INFO SparkContext - Starting job: collect at SparkSharder.java:388
19:01:59.799 INFO DAGScheduler - Got job 8 (collect at SparkSharder.java:388) with 2 output partitions
19:01:59.799 INFO DAGScheduler - Final stage: ResultStage 12 (collect at SparkSharder.java:388)
19:01:59.799 INFO DAGScheduler - Parents of final stage: List()
19:01:59.799 INFO DAGScheduler - Missing parents: List()
19:01:59.800 INFO DAGScheduler - Submitting ResultStage 12 (MapPartitionsRDD[28] at mapPartitions at SparkSharder.java:386), which has no missing parents
19:01:59.802 INFO MemoryStore - Block broadcast_14 stored as values in memory (estimated size 4.5 KiB, free 1919.9 MiB)
19:01:59.804 INFO MemoryStore - Block broadcast_14_piece0 stored as bytes in memory (estimated size 2.4 KiB, free 1919.9 MiB)
19:01:59.805 INFO BlockManagerInfo - Added broadcast_14_piece0 in memory on localhost:45255 (size: 2.4 KiB, free: 1920.0 MiB)
19:01:59.806 INFO SparkContext - Created broadcast 14 from broadcast at DAGScheduler.scala:1580
19:01:59.806 INFO DAGScheduler - Submitting 2 missing tasks from ResultStage 12 (MapPartitionsRDD[28] at mapPartitions at SparkSharder.java:386) (first 15 tasks are for partitions Vector(0, 1))
19:01:59.807 INFO TaskSchedulerImpl - Adding task set 12.0 with 2 tasks resource profile 0
19:01:59.808 INFO TaskSetManager - Starting task 0.0 in stage 12.0 (TID 17) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7692 bytes)
19:01:59.809 INFO TaskSetManager - Starting task 1.0 in stage 12.0 (TID 18) (localhost, executor driver, partition 1, PROCESS_LOCAL, 7692 bytes)
19:01:59.809 INFO Executor - Running task 0.0 in stage 12.0 (TID 17)
19:01:59.810 INFO Executor - Running task 1.0 in stage 12.0 (TID 18)
19:01:59.814 INFO Executor - Finished task 1.0 in stage 12.0 (TID 18). 1073 bytes result sent to driver
19:01:59.815 INFO TaskSetManager - Finished task 1.0 in stage 12.0 (TID 18) in 7 ms on localhost (executor driver) (1/2)
19:01:59.815 INFO Executor - Finished task 0.0 in stage 12.0 (TID 17). 1073 bytes result sent to driver
19:01:59.816 INFO TaskSetManager - Finished task 0.0 in stage 12.0 (TID 17) in 8 ms on localhost (executor driver) (2/2)
19:01:59.816 INFO TaskSchedulerImpl - Removed TaskSet 12.0, whose tasks have all completed, from pool
19:01:59.817 INFO DAGScheduler - ResultStage 12 (collect at SparkSharder.java:388) finished in 0.016 s
19:01:59.817 INFO DAGScheduler - Job 8 is finished. Cancelling potential speculative or zombie tasks for this job
19:01:59.817 INFO TaskSchedulerImpl - Killing all running tasks in stage 12: Stage finished
19:01:59.817 INFO DAGScheduler - Job 8 finished: collect at SparkSharder.java:388, took 0.019358 s
19:01:59.832 INFO SparkContext - Starting job: collect at SparkSharder.java:388
19:01:59.833 INFO DAGScheduler - Got job 9 (collect at SparkSharder.java:388) with 1 output partitions
19:01:59.833 INFO DAGScheduler - Final stage: ResultStage 13 (collect at SparkSharder.java:388)
19:01:59.833 INFO DAGScheduler - Parents of final stage: List()
19:01:59.833 INFO DAGScheduler - Missing parents: List()
19:01:59.833 INFO DAGScheduler - Submitting ResultStage 13 (MapPartitionsRDD[30] at mapPartitions at SparkSharder.java:386), which has no missing parents
19:01:59.836 INFO MemoryStore - Block broadcast_15 stored as values in memory (estimated size 4.5 KiB, free 1919.9 MiB)
19:01:59.837 INFO MemoryStore - Block broadcast_15_piece0 stored as bytes in memory (estimated size 2.4 KiB, free 1919.9 MiB)
19:01:59.838 INFO BlockManagerInfo - Added broadcast_15_piece0 in memory on localhost:45255 (size: 2.4 KiB, free: 1920.0 MiB)
19:01:59.838 INFO SparkContext - Created broadcast 15 from broadcast at DAGScheduler.scala:1580
19:01:59.839 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 13 (MapPartitionsRDD[30] at mapPartitions at SparkSharder.java:386) (first 15 tasks are for partitions Vector(0))
19:01:59.839 INFO TaskSchedulerImpl - Adding task set 13.0 with 1 tasks resource profile 0
19:01:59.840 INFO TaskSetManager - Starting task 0.0 in stage 13.0 (TID 19) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7730 bytes)
19:01:59.841 INFO Executor - Running task 0.0 in stage 13.0 (TID 19)
19:01:59.845 INFO Executor - Finished task 0.0 in stage 13.0 (TID 19). 1073 bytes result sent to driver
19:01:59.846 INFO TaskSetManager - Finished task 0.0 in stage 13.0 (TID 19) in 6 ms on localhost (executor driver) (1/1)
19:01:59.847 INFO TaskSchedulerImpl - Removed TaskSet 13.0, whose tasks have all completed, from pool
19:01:59.847 INFO DAGScheduler - ResultStage 13 (collect at SparkSharder.java:388) finished in 0.013 s
19:01:59.847 INFO DAGScheduler - Job 9 is finished. Cancelling potential speculative or zombie tasks for this job
19:01:59.848 INFO TaskSchedulerImpl - Killing all running tasks in stage 13: Stage finished
19:01:59.848 INFO DAGScheduler - Job 9 finished: collect at SparkSharder.java:388, took 0.015411 s
19:01:59.863 INFO SparkContext - Starting job: collect at SparkSharder.java:388
19:01:59.863 INFO DAGScheduler - Got job 10 (collect at SparkSharder.java:388) with 2 output partitions
19:01:59.863 INFO DAGScheduler - Final stage: ResultStage 14 (collect at SparkSharder.java:388)
19:01:59.864 INFO DAGScheduler - Parents of final stage: List()
19:01:59.864 INFO DAGScheduler - Missing parents: List()
19:01:59.864 INFO DAGScheduler - Submitting ResultStage 14 (MapPartitionsRDD[32] at mapPartitions at SparkSharder.java:386), which has no missing parents
19:01:59.865 INFO MemoryStore - Block broadcast_16 stored as values in memory (estimated size 4.5 KiB, free 1919.9 MiB)
19:01:59.867 INFO MemoryStore - Block broadcast_16_piece0 stored as bytes in memory (estimated size 2.4 KiB, free 1919.9 MiB)
19:01:59.868 INFO BlockManagerInfo - Added broadcast_16_piece0 in memory on localhost:45255 (size: 2.4 KiB, free: 1920.0 MiB)
19:01:59.868 INFO SparkContext - Created broadcast 16 from broadcast at DAGScheduler.scala:1580
19:01:59.869 INFO DAGScheduler - Submitting 2 missing tasks from ResultStage 14 (MapPartitionsRDD[32] at mapPartitions at SparkSharder.java:386) (first 15 tasks are for partitions Vector(0, 1))
19:01:59.869 INFO TaskSchedulerImpl - Adding task set 14.0 with 2 tasks resource profile 0
19:01:59.870 INFO TaskSetManager - Starting task 0.0 in stage 14.0 (TID 20) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7692 bytes)
19:01:59.871 INFO TaskSetManager - Starting task 1.0 in stage 14.0 (TID 21) (localhost, executor driver, partition 1, PROCESS_LOCAL, 7692 bytes)
19:01:59.871 INFO Executor - Running task 1.0 in stage 14.0 (TID 21)
19:01:59.872 INFO Executor - Running task 0.0 in stage 14.0 (TID 20)
19:01:59.876 INFO Executor - Finished task 0.0 in stage 14.0 (TID 20). 1073 bytes result sent to driver
19:01:59.877 INFO Executor - Finished task 1.0 in stage 14.0 (TID 21). 1073 bytes result sent to driver
19:01:59.877 INFO TaskSetManager - Finished task 0.0 in stage 14.0 (TID 20) in 7 ms on localhost (executor driver) (1/2)
19:01:59.878 INFO TaskSetManager - Finished task 1.0 in stage 14.0 (TID 21) in 8 ms on localhost (executor driver) (2/2)
19:01:59.878 INFO TaskSchedulerImpl - Removed TaskSet 14.0, whose tasks have all completed, from pool
19:01:59.879 INFO DAGScheduler - ResultStage 14 (collect at SparkSharder.java:388) finished in 0.015 s
19:01:59.879 INFO DAGScheduler - Job 10 is finished. Cancelling potential speculative or zombie tasks for this job
19:01:59.879 INFO TaskSchedulerImpl - Killing all running tasks in stage 14: Stage finished
19:01:59.879 INFO DAGScheduler - Job 10 finished: collect at SparkSharder.java:388, took 0.016302 s
19:01:59.893 INFO SparkContext - Starting job: collect at SparkSharder.java:388
19:01:59.893 INFO DAGScheduler - Got job 11 (collect at SparkSharder.java:388) with 3 output partitions
19:01:59.893 INFO DAGScheduler - Final stage: ResultStage 15 (collect at SparkSharder.java:388)
19:01:59.893 INFO DAGScheduler - Parents of final stage: List()
19:01:59.893 INFO DAGScheduler - Missing parents: List()
19:01:59.894 INFO DAGScheduler - Submitting ResultStage 15 (MapPartitionsRDD[34] at mapPartitions at SparkSharder.java:386), which has no missing parents
19:01:59.895 INFO MemoryStore - Block broadcast_17 stored as values in memory (estimated size 4.5 KiB, free 1919.9 MiB)
19:01:59.896 INFO MemoryStore - Block broadcast_17_piece0 stored as bytes in memory (estimated size 2.4 KiB, free 1919.9 MiB)
19:01:59.896 INFO BlockManagerInfo - Added broadcast_17_piece0 in memory on localhost:45255 (size: 2.4 KiB, free: 1920.0 MiB)
19:01:59.896 INFO SparkContext - Created broadcast 17 from broadcast at DAGScheduler.scala:1580
19:01:59.897 INFO DAGScheduler - Submitting 3 missing tasks from ResultStage 15 (MapPartitionsRDD[34] at mapPartitions at SparkSharder.java:386) (first 15 tasks are for partitions Vector(0, 1, 2))
19:01:59.897 INFO TaskSchedulerImpl - Adding task set 15.0 with 3 tasks resource profile 0
19:01:59.898 INFO TaskSetManager - Starting task 0.0 in stage 15.0 (TID 22) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7680 bytes)
19:01:59.898 INFO TaskSetManager - Starting task 1.0 in stage 15.0 (TID 23) (localhost, executor driver, partition 1, PROCESS_LOCAL, 7682 bytes)
19:01:59.898 INFO TaskSetManager - Starting task 2.0 in stage 15.0 (TID 24) (localhost, executor driver, partition 2, PROCESS_LOCAL, 7680 bytes)
19:01:59.899 INFO Executor - Running task 0.0 in stage 15.0 (TID 22)
19:01:59.901 INFO Executor - Running task 2.0 in stage 15.0 (TID 24)
19:01:59.902 INFO Executor - Running task 1.0 in stage 15.0 (TID 23)
19:01:59.904 INFO Executor - Finished task 1.0 in stage 15.0 (TID 23). 1030 bytes result sent to driver
19:01:59.905 INFO TaskSetManager - Finished task 1.0 in stage 15.0 (TID 23) in 7 ms on localhost (executor driver) (1/3)
19:01:59.906 INFO Executor - Finished task 2.0 in stage 15.0 (TID 24). 1073 bytes result sent to driver
19:01:59.906 INFO Executor - Finished task 0.0 in stage 15.0 (TID 22). 1073 bytes result sent to driver
19:01:59.907 INFO TaskSetManager - Finished task 2.0 in stage 15.0 (TID 24) in 9 ms on localhost (executor driver) (2/3)
19:01:59.908 INFO TaskSetManager - Finished task 0.0 in stage 15.0 (TID 22) in 11 ms on localhost (executor driver) (3/3)
19:01:59.908 INFO TaskSchedulerImpl - Removed TaskSet 15.0, whose tasks have all completed, from pool
19:01:59.908 INFO DAGScheduler - ResultStage 15 (collect at SparkSharder.java:388) finished in 0.014 s
19:01:59.908 INFO DAGScheduler - Job 11 is finished. Cancelling potential speculative or zombie tasks for this job
19:01:59.908 INFO TaskSchedulerImpl - Killing all running tasks in stage 15: Stage finished
19:01:59.908 INFO DAGScheduler - Job 11 finished: collect at SparkSharder.java:388, took 0.015551 s
19:01:59.923 INFO SparkContext - Starting job: collect at SparkSharder.java:388
19:01:59.924 INFO DAGScheduler - Got job 12 (collect at SparkSharder.java:388) with 1 output partitions
19:01:59.924 INFO DAGScheduler - Final stage: ResultStage 16 (collect at SparkSharder.java:388)
19:01:59.924 INFO DAGScheduler - Parents of final stage: List()
19:01:59.924 INFO DAGScheduler - Missing parents: List()
19:01:59.924 INFO DAGScheduler - Submitting ResultStage 16 (MapPartitionsRDD[36] at mapPartitions at SparkSharder.java:386), which has no missing parents
19:01:59.926 INFO MemoryStore - Block broadcast_18 stored as values in memory (estimated size 4.5 KiB, free 1919.9 MiB)
19:01:59.927 INFO MemoryStore - Block broadcast_18_piece0 stored as bytes in memory (estimated size 2.4 KiB, free 1919.9 MiB)
19:01:59.928 INFO BlockManagerInfo - Added broadcast_18_piece0 in memory on localhost:45255 (size: 2.4 KiB, free: 1920.0 MiB)
19:01:59.928 INFO SparkContext - Created broadcast 18 from broadcast at DAGScheduler.scala:1580
19:01:59.929 INFO DAGScheduler - Submitting 1 missing tasks from ResultStage 16 (MapPartitionsRDD[36] at mapPartitions at SparkSharder.java:386) (first 15 tasks are for partitions Vector(0))
19:01:59.929 INFO TaskSchedulerImpl - Adding task set 16.0 with 1 tasks resource profile 0
19:01:59.930 INFO TaskSetManager - Starting task 0.0 in stage 16.0 (TID 25) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7730 bytes)
19:01:59.930 INFO Executor - Running task 0.0 in stage 16.0 (TID 25)
19:01:59.934 INFO Executor - Finished task 0.0 in stage 16.0 (TID 25). 1030 bytes result sent to driver
19:01:59.934 INFO TaskSetManager - Finished task 0.0 in stage 16.0 (TID 25) in 4 ms on localhost (executor driver) (1/1)
19:01:59.934 INFO TaskSchedulerImpl - Removed TaskSet 16.0, whose tasks have all completed, from pool
19:01:59.935 INFO DAGScheduler - ResultStage 16 (collect at SparkSharder.java:388) finished in 0.010 s
19:01:59.935 INFO DAGScheduler - Job 12 is finished. Cancelling potential speculative or zombie tasks for this job
19:01:59.935 INFO TaskSchedulerImpl - Killing all running tasks in stage 16: Stage finished
19:01:59.935 INFO DAGScheduler - Job 12 finished: collect at SparkSharder.java:388, took 0.012289 s
19:01:59.949 INFO SparkContext - Starting job: collect at SparkSharder.java:388
19:01:59.949 INFO DAGScheduler - Got job 13 (collect at SparkSharder.java:388) with 2 output partitions
19:01:59.949 INFO DAGScheduler - Final stage: ResultStage 17 (collect at SparkSharder.java:388)
19:01:59.949 INFO DAGScheduler - Parents of final stage: List()
19:01:59.949 INFO DAGScheduler - Missing parents: List()
19:01:59.950 INFO DAGScheduler - Submitting ResultStage 17 (MapPartitionsRDD[38] at mapPartitions at SparkSharder.java:386), which has no missing parents
19:01:59.952 INFO MemoryStore - Block broadcast_19 stored as values in memory (estimated size 4.5 KiB, free 1919.9 MiB)
19:01:59.953 INFO MemoryStore - Block broadcast_19_piece0 stored as bytes in memory (estimated size 2.4 KiB, free 1919.9 MiB)
19:01:59.953 INFO BlockManagerInfo - Added broadcast_19_piece0 in memory on localhost:45255 (size: 2.4 KiB, free: 1919.9 MiB)
19:01:59.954 INFO SparkContext - Created broadcast 19 from broadcast at DAGScheduler.scala:1580
19:01:59.954 INFO DAGScheduler - Submitting 2 missing tasks from ResultStage 17 (MapPartitionsRDD[38] at mapPartitions at SparkSharder.java:386) (first 15 tasks are for partitions Vector(0, 1))
19:01:59.954 INFO TaskSchedulerImpl - Adding task set 17.0 with 2 tasks resource profile 0
19:01:59.955 INFO TaskSetManager - Starting task 0.0 in stage 17.0 (TID 26) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7692 bytes)
19:01:59.956 INFO TaskSetManager - Starting task 1.0 in stage 17.0 (TID 27) (localhost, executor driver, partition 1, PROCESS_LOCAL, 7692 bytes)
19:01:59.956 INFO Executor - Running task 1.0 in stage 17.0 (TID 27)
19:01:59.959 INFO Executor - Running task 0.0 in stage 17.0 (TID 26)
19:01:59.961 INFO Executor - Finished task 1.0 in stage 17.0 (TID 27). 1030 bytes result sent to driver
19:01:59.962 INFO TaskSetManager - Finished task 1.0 in stage 17.0 (TID 27) in 7 ms on localhost (executor driver) (1/2)
19:01:59.964 INFO Executor - Finished task 0.0 in stage 17.0 (TID 26). 1030 bytes result sent to driver
19:01:59.965 INFO TaskSetManager - Finished task 0.0 in stage 17.0 (TID 26) in 10 ms on localhost (executor driver) (2/2)
19:01:59.965 INFO DAGScheduler - ResultStage 17 (collect at SparkSharder.java:388) finished in 0.014 s
19:01:59.965 INFO DAGScheduler - Job 13 is finished. Cancelling potential speculative or zombie tasks for this job
19:01:59.966 INFO TaskSchedulerImpl - Removed TaskSet 17.0, whose tasks have all completed, from pool
19:01:59.966 INFO TaskSchedulerImpl - Killing all running tasks in stage 17: Stage finished
19:01:59.966 INFO DAGScheduler - Job 13 finished: collect at SparkSharder.java:388, took 0.017348 s
19:01:59.980 INFO SparkContext - Starting job: collect at SparkSharder.java:388
19:01:59.981 INFO DAGScheduler - Got job 14 (collect at SparkSharder.java:388) with 3 output partitions
19:01:59.981 INFO DAGScheduler - Final stage: ResultStage 18 (collect at SparkSharder.java:388)
19:01:59.981 INFO DAGScheduler - Parents of final stage: List()
19:01:59.981 INFO DAGScheduler - Missing parents: List()
19:01:59.982 INFO DAGScheduler - Submitting ResultStage 18 (MapPartitionsRDD[40] at mapPartitions at SparkSharder.java:386), which has no missing parents
19:01:59.983 INFO MemoryStore - Block broadcast_20 stored as values in memory (estimated size 4.5 KiB, free 1919.8 MiB)
19:01:59.984 INFO MemoryStore - Block broadcast_20_piece0 stored as bytes in memory (estimated size 2.4 KiB, free 1919.8 MiB)
19:01:59.985 INFO BlockManagerInfo - Added broadcast_20_piece0 in memory on localhost:45255 (size: 2.4 KiB, free: 1919.9 MiB)
19:01:59.985 INFO SparkContext - Created broadcast 20 from broadcast at DAGScheduler.scala:1580
19:01:59.985 INFO DAGScheduler - Submitting 3 missing tasks from ResultStage 18 (MapPartitionsRDD[40] at mapPartitions at SparkSharder.java:386) (first 15 tasks are for partitions Vector(0, 1, 2))
19:01:59.985 INFO TaskSchedulerImpl - Adding task set 18.0 with 3 tasks resource profile 0
19:01:59.986 INFO TaskSetManager - Starting task 0.0 in stage 18.0 (TID 28) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7680 bytes)
19:01:59.987 INFO TaskSetManager - Starting task 1.0 in stage 18.0 (TID 29) (localhost, executor driver, partition 1, PROCESS_LOCAL, 7682 bytes)
19:01:59.987 INFO TaskSetManager - Starting task 2.0 in stage 18.0 (TID 30) (localhost, executor driver, partition 2, PROCESS_LOCAL, 7680 bytes)
19:01:59.987 INFO Executor - Running task 1.0 in stage 18.0 (TID 29)
19:01:59.988 INFO Executor - Running task 0.0 in stage 18.0 (TID 28)
19:01:59.990 INFO Executor - Running task 2.0 in stage 18.0 (TID 30)
19:01:59.993 INFO Executor - Finished task 0.0 in stage 18.0 (TID 28). 1073 bytes result sent to driver
19:01:59.994 INFO Executor - Finished task 2.0 in stage 18.0 (TID 30). 1073 bytes result sent to driver
19:01:59.995 INFO Executor - Finished task 1.0 in stage 18.0 (TID 29). 1073 bytes result sent to driver
19:01:59.996 INFO TaskSetManager - Finished task 0.0 in stage 18.0 (TID 28) in 10 ms on localhost (executor driver) (1/3)
19:01:59.996 INFO TaskSetManager - Finished task 2.0 in stage 18.0 (TID 30) in 9 ms on localhost (executor driver) (2/3)
19:01:59.996 INFO TaskSetManager - Finished task 1.0 in stage 18.0 (TID 29) in 10 ms on localhost (executor driver) (3/3)
19:01:59.996 INFO TaskSchedulerImpl - Removed TaskSet 18.0, whose tasks have all completed, from pool
19:01:59.997 INFO DAGScheduler - ResultStage 18 (collect at SparkSharder.java:388) finished in 0.015 s
19:01:59.997 INFO DAGScheduler - Job 14 is finished. Cancelling potential speculative or zombie tasks for this job
19:01:59.997 INFO TaskSchedulerImpl - Killing all running tasks in stage 18: Stage finished
19:01:59.998 INFO DAGScheduler - Job 14 finished: collect at SparkSharder.java:388, took 0.017363 s
19:02:00.014 INFO SparkContext - Starting job: collect at SparkSharder.java:388
19:02:00.014 INFO DAGScheduler - Got job 15 (collect at SparkSharder.java:388) with 4 output partitions
19:02:00.014 INFO DAGScheduler - Final stage: ResultStage 19 (collect at SparkSharder.java:388)
19:02:00.015 INFO DAGScheduler - Parents of final stage: List()
19:02:00.015 INFO DAGScheduler - Missing parents: List()
19:02:00.016 INFO DAGScheduler - Submitting ResultStage 19 (MapPartitionsRDD[43] at mapPartitions at SparkSharder.java:386), which has no missing parents
19:02:00.017 INFO MemoryStore - Block broadcast_21 stored as values in memory (estimated size 4.5 KiB, free 1919.8 MiB)
19:02:00.018 INFO MemoryStore - Block broadcast_21_piece0 stored as bytes in memory (estimated size 2.4 KiB, free 1919.8 MiB)
19:02:00.019 INFO BlockManagerInfo - Added broadcast_21_piece0 in memory on localhost:45255 (size: 2.4 KiB, free: 1919.9 MiB)
19:02:00.019 INFO SparkContext - Created broadcast 21 from broadcast at DAGScheduler.scala:1580
19:02:00.020 INFO DAGScheduler - Submitting 4 missing tasks from ResultStage 19 (MapPartitionsRDD[43] at mapPartitions at SparkSharder.java:386) (first 15 tasks are for partitions Vector(0, 1, 2, 3))
19:02:00.020 INFO TaskSchedulerImpl - Adding task set 19.0 with 4 tasks resource profile 0
19:02:00.021 INFO TaskSetManager - Starting task 0.0 in stage 19.0 (TID 31) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7692 bytes)
19:02:00.021 INFO TaskSetManager - Starting task 1.0 in stage 19.0 (TID 32) (localhost, executor driver, partition 1, PROCESS_LOCAL, 7692 bytes)
19:02:00.022 INFO TaskSetManager - Starting task 2.0 in stage 19.0 (TID 33) (localhost, executor driver, partition 2, PROCESS_LOCAL, 7692 bytes)
19:02:00.022 INFO TaskSetManager - Starting task 3.0 in stage 19.0 (TID 34) (localhost, executor driver, partition 3, PROCESS_LOCAL, 7692 bytes)
19:02:00.022 INFO Executor - Running task 2.0 in stage 19.0 (TID 33)
19:02:00.023 INFO Executor - Running task 1.0 in stage 19.0 (TID 32)
19:02:00.022 INFO Executor - Running task 3.0 in stage 19.0 (TID 34)
19:02:00.025 INFO Executor - Running task 0.0 in stage 19.0 (TID 31)
19:02:00.030 INFO Executor - Finished task 1.0 in stage 19.0 (TID 32). 1073 bytes result sent to driver
19:02:00.031 INFO TaskSetManager - Finished task 1.0 in stage 19.0 (TID 32) in 10 ms on localhost (executor driver) (1/4)
19:02:00.031 INFO Executor - Finished task 0.0 in stage 19.0 (TID 31). 1073 bytes result sent to driver
19:02:00.032 INFO Executor - Finished task 2.0 in stage 19.0 (TID 33). 1030 bytes result sent to driver
19:02:00.032 INFO Executor - Finished task 3.0 in stage 19.0 (TID 34). 1030 bytes result sent to driver
19:02:00.034 INFO TaskSetManager - Finished task 2.0 in stage 19.0 (TID 33) in 13 ms on localhost (executor driver) (2/4)
19:02:00.035 INFO TaskSetManager - Finished task 0.0 in stage 19.0 (TID 31) in 15 ms on localhost (executor driver) (3/4)
19:02:00.036 INFO TaskSetManager - Finished task 3.0 in stage 19.0 (TID 34) in 14 ms on localhost (executor driver) (4/4)
19:02:00.036 INFO TaskSchedulerImpl - Removed TaskSet 19.0, whose tasks have all completed, from pool
19:02:00.037 INFO DAGScheduler - ResultStage 19 (collect at SparkSharder.java:388) finished in 0.020 s
19:02:00.037 INFO DAGScheduler - Job 15 is finished. Cancelling potential speculative or zombie tasks for this job
19:02:00.037 INFO TaskSchedulerImpl - Killing all running tasks in stage 19: Stage finished
19:02:00.037 INFO DAGScheduler - Job 15 finished: collect at SparkSharder.java:388, took 0.023323 s
19:02:00.040 INFO MemoryStore - Block broadcast_22 stored as values in memory (estimated size 464.0 B, free 1919.8 MiB)
19:02:00.041 INFO MemoryStore - Block broadcast_22_piece0 stored as bytes in memory (estimated size 159.0 B, free 1919.8 MiB)
19:02:00.041 INFO BlockManagerInfo - Added broadcast_22_piece0 in memory on localhost:45255 (size: 159.0 B, free: 1919.9 MiB)
19:02:00.042 INFO SparkContext - Created broadcast 22 from broadcast at SparkSharder.java:214
19:02:00.042 INFO MemoryStore - Block broadcast_23 stored as values in memory (estimated size 1088.0 B, free 1919.8 MiB)
19:02:00.044 INFO MemoryStore - Block broadcast_23_piece0 stored as bytes in memory (estimated size 465.0 B, free 1919.8 MiB)
19:02:00.044 INFO BlockManagerInfo - Added broadcast_23_piece0 in memory on localhost:45255 (size: 465.0 B, free: 1919.9 MiB)
19:02:00.045 INFO SparkContext - Created broadcast 23 from broadcast at SparkSharder.java:221
19:02:00.065 INFO SparkContext - Starting job: collectAsMap at SparkSharder.java:258
19:02:00.066 INFO DAGScheduler - Registering RDD 45 (mapToPair at SparkSharder.java:247) as input to shuffle 4
19:02:00.066 INFO DAGScheduler - Registering RDD 48 (mapToPair at SparkSharder.java:255) as input to shuffle 3
19:02:00.067 INFO DAGScheduler - Got job 16 (collectAsMap at SparkSharder.java:258) with 4 output partitions
19:02:00.067 INFO DAGScheduler - Final stage: ResultStage 22 (collectAsMap at SparkSharder.java:258)
19:02:00.067 INFO DAGScheduler - Parents of final stage: List(ShuffleMapStage 21)
19:02:00.067 INFO DAGScheduler - Missing parents: List(ShuffleMapStage 21)
19:02:00.069 INFO DAGScheduler - Submitting ShuffleMapStage 20 (MapPartitionsRDD[45] at mapToPair at SparkSharder.java:247), which has no missing parents
19:02:00.070 INFO MemoryStore - Block broadcast_24 stored as values in memory (estimated size 7.9 KiB, free 1919.8 MiB)
19:02:00.072 INFO MemoryStore - Block broadcast_24_piece0 stored as bytes in memory (estimated size 4.3 KiB, free 1919.8 MiB)
19:02:00.072 INFO BlockManagerInfo - Added broadcast_24_piece0 in memory on localhost:45255 (size: 4.3 KiB, free: 1919.9 MiB)
19:02:00.072 INFO SparkContext - Created broadcast 24 from broadcast at DAGScheduler.scala:1580
19:02:00.072 INFO DAGScheduler - Submitting 4 missing tasks from ShuffleMapStage 20 (MapPartitionsRDD[45] at mapToPair at SparkSharder.java:247) (first 15 tasks are for partitions Vector(0, 1, 2, 3))
19:02:00.073 INFO TaskSchedulerImpl - Adding task set 20.0 with 4 tasks resource profile 0
19:02:00.074 INFO TaskSetManager - Starting task 0.0 in stage 20.0 (TID 35) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7569 bytes)
19:02:00.074 INFO TaskSetManager - Starting task 1.0 in stage 20.0 (TID 36) (localhost, executor driver, partition 1, PROCESS_LOCAL, 7636 bytes)
19:02:00.074 INFO TaskSetManager - Starting task 2.0 in stage 20.0 (TID 37) (localhost, executor driver, partition 2, PROCESS_LOCAL, 7636 bytes)
19:02:00.075 INFO TaskSetManager - Starting task 3.0 in stage 20.0 (TID 38) (localhost, executor driver, partition 3, PROCESS_LOCAL, 7636 bytes)
19:02:00.076 INFO Executor - Running task 1.0 in stage 20.0 (TID 36)
19:02:00.076 INFO Executor - Running task 2.0 in stage 20.0 (TID 37)
19:02:00.076 INFO Executor - Running task 3.0 in stage 20.0 (TID 38)
19:02:00.076 INFO Executor - Running task 0.0 in stage 20.0 (TID 35)
19:02:00.087 INFO Executor - Finished task 0.0 in stage 20.0 (TID 35). 936 bytes result sent to driver
19:02:00.088 INFO TaskSetManager - Finished task 0.0 in stage 20.0 (TID 35) in 15 ms on localhost (executor driver) (1/4)
19:02:00.093 INFO Executor - Finished task 3.0 in stage 20.0 (TID 38). 1065 bytes result sent to driver
19:02:00.094 INFO TaskSetManager - Finished task 3.0 in stage 20.0 (TID 38) in 19 ms on localhost (executor driver) (2/4)
19:02:00.095 INFO Executor - Finished task 1.0 in stage 20.0 (TID 36). 1065 bytes result sent to driver
19:02:00.095 INFO TaskSetManager - Finished task 1.0 in stage 20.0 (TID 36) in 21 ms on localhost (executor driver) (3/4)
19:02:00.097 INFO Executor - Finished task 2.0 in stage 20.0 (TID 37). 1065 bytes result sent to driver
19:02:00.097 INFO TaskSetManager - Finished task 2.0 in stage 20.0 (TID 37) in 23 ms on localhost (executor driver) (4/4)
19:02:00.097 INFO TaskSchedulerImpl - Removed TaskSet 20.0, whose tasks have all completed, from pool
19:02:00.097 INFO DAGScheduler - ShuffleMapStage 20 (mapToPair at SparkSharder.java:247) finished in 0.028 s
19:02:00.098 INFO DAGScheduler - looking for newly runnable stages
19:02:00.098 INFO DAGScheduler - running: HashSet()
19:02:00.098 INFO DAGScheduler - waiting: HashSet(ShuffleMapStage 21, ResultStage 22)
19:02:00.098 INFO DAGScheduler - failed: HashSet()
19:02:00.098 INFO DAGScheduler - Submitting ShuffleMapStage 21 (MapPartitionsRDD[48] at mapToPair at SparkSharder.java:255), which has no missing parents
19:02:00.100 INFO MemoryStore - Block broadcast_25 stored as values in memory (estimated size 8.2 KiB, free 1919.8 MiB)
19:02:00.101 INFO MemoryStore - Block broadcast_25_piece0 stored as bytes in memory (estimated size 4.5 KiB, free 1919.8 MiB)
19:02:00.101 INFO BlockManagerInfo - Added broadcast_25_piece0 in memory on localhost:45255 (size: 4.5 KiB, free: 1919.9 MiB)
19:02:00.101 INFO SparkContext - Created broadcast 25 from broadcast at DAGScheduler.scala:1580
19:02:00.102 INFO DAGScheduler - Submitting 4 missing tasks from ShuffleMapStage 21 (MapPartitionsRDD[48] at mapToPair at SparkSharder.java:255) (first 15 tasks are for partitions Vector(0, 1, 2, 3))
19:02:00.102 INFO TaskSchedulerImpl - Adding task set 21.0 with 4 tasks resource profile 0
19:02:00.103 INFO TaskSetManager - Starting task 0.0 in stage 21.0 (TID 39) (localhost, executor driver, partition 0, NODE_LOCAL, 7502 bytes)
19:02:00.103 INFO TaskSetManager - Starting task 2.0 in stage 21.0 (TID 40) (localhost, executor driver, partition 2, NODE_LOCAL, 7502 bytes)
19:02:00.104 INFO TaskSetManager - Starting task 1.0 in stage 21.0 (TID 41) (localhost, executor driver, partition 1, PROCESS_LOCAL, 7502 bytes)
19:02:00.104 INFO TaskSetManager - Starting task 3.0 in stage 21.0 (TID 42) (localhost, executor driver, partition 3, PROCESS_LOCAL, 7502 bytes)
19:02:00.105 INFO Executor - Running task 3.0 in stage 21.0 (TID 42)
19:02:00.105 INFO Executor - Running task 0.0 in stage 21.0 (TID 39)
19:02:00.105 INFO Executor - Running task 1.0 in stage 21.0 (TID 41)
19:02:00.110 INFO Executor - Running task 2.0 in stage 21.0 (TID 40)
19:02:00.110 INFO ShuffleBlockFetcherIterator - Getting 2 (342.0 B) non-empty blocks including 2 (342.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
19:02:00.111 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
19:02:00.114 INFO ShuffleBlockFetcherIterator - Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
19:02:00.114 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
19:02:00.114 INFO MemoryStore - Block rdd_47_1 stored as values in memory (estimated size 16.0 B, free 1919.8 MiB)
19:02:00.114 INFO BlockManagerInfo - Added rdd_47_1 in memory on localhost:45255 (size: 16.0 B, free: 1919.9 MiB)
19:02:00.115 INFO ShuffleBlockFetcherIterator - Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
19:02:00.115 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
19:02:00.116 INFO MemoryStore - Block rdd_47_3 stored as values in memory (estimated size 16.0 B, free 1919.8 MiB)
19:02:00.117 INFO ShuffleBlockFetcherIterator - Getting 1 (171.0 B) non-empty blocks including 1 (171.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
19:02:00.117 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
19:02:00.119 INFO BlockManagerInfo - Added rdd_47_3 in memory on localhost:45255 (size: 16.0 B, free: 1919.9 MiB)
19:02:00.120 INFO Executor - Finished task 1.0 in stage 21.0 (TID 41). 1839 bytes result sent to driver
19:02:00.121 INFO TaskSetManager - Finished task 1.0 in stage 21.0 (TID 41) in 17 ms on localhost (executor driver) (1/4)
19:02:00.124 INFO Executor - Finished task 3.0 in stage 21.0 (TID 42). 1839 bytes result sent to driver
19:02:00.126 INFO MemoryStore - Block rdd_47_0 stored as values in memory (estimated size 264.0 B, free 1919.8 MiB)
19:02:00.126 INFO BlockManagerInfo - Added rdd_47_0 in memory on localhost:45255 (size: 264.0 B, free: 1919.9 MiB)
19:02:00.126 INFO TaskSetManager - Finished task 3.0 in stage 21.0 (TID 42) in 22 ms on localhost (executor driver) (2/4)
19:02:00.127 INFO MemoryStore - Block rdd_47_2 stored as values in memory (estimated size 144.0 B, free 1919.8 MiB)
19:02:00.129 INFO Executor - Finished task 0.0 in stage 21.0 (TID 39). 1925 bytes result sent to driver
19:02:00.130 INFO BlockManagerInfo - Added rdd_47_2 in memory on localhost:45255 (size: 144.0 B, free: 1919.9 MiB)
19:02:00.131 INFO TaskSetManager - Finished task 0.0 in stage 21.0 (TID 39) in 28 ms on localhost (executor driver) (3/4)
19:02:00.132 INFO Executor - Finished task 2.0 in stage 21.0 (TID 40). 1925 bytes result sent to driver
19:02:00.133 INFO TaskSetManager - Finished task 2.0 in stage 21.0 (TID 40) in 30 ms on localhost (executor driver) (4/4)
19:02:00.133 INFO DAGScheduler - ShuffleMapStage 21 (mapToPair at SparkSharder.java:255) finished in 0.034 s
19:02:00.133 INFO DAGScheduler - looking for newly runnable stages
19:02:00.133 INFO DAGScheduler - running: HashSet()
19:02:00.133 INFO DAGScheduler - waiting: HashSet(ResultStage 22)
19:02:00.133 INFO DAGScheduler - failed: HashSet()
19:02:00.134 INFO DAGScheduler - Submitting ResultStage 22 (ShuffledRDD[49] at reduceByKey at SparkSharder.java:257), which has no missing parents
19:02:00.134 INFO TaskSchedulerImpl - Removed TaskSet 21.0, whose tasks have all completed, from pool
19:02:00.135 INFO MemoryStore - Block broadcast_26 stored as values in memory (estimated size 5.8 KiB, free 1919.8 MiB)
19:02:00.136 INFO MemoryStore - Block broadcast_26_piece0 stored as bytes in memory (estimated size 3.2 KiB, free 1919.8 MiB)
19:02:00.136 INFO BlockManagerInfo - Added broadcast_26_piece0 in memory on localhost:45255 (size: 3.2 KiB, free: 1919.9 MiB)
19:02:00.136 INFO SparkContext - Created broadcast 26 from broadcast at DAGScheduler.scala:1580
19:02:00.137 INFO DAGScheduler - Submitting 4 missing tasks from ResultStage 22 (ShuffledRDD[49] at reduceByKey at SparkSharder.java:257) (first 15 tasks are for partitions Vector(0, 1, 2, 3))
19:02:00.137 INFO TaskSchedulerImpl - Adding task set 22.0 with 4 tasks resource profile 0
19:02:00.138 INFO TaskSetManager - Starting task 0.0 in stage 22.0 (TID 43) (localhost, executor driver, partition 0, NODE_LOCAL, 7513 bytes)
19:02:00.138 INFO TaskSetManager - Starting task 2.0 in stage 22.0 (TID 44) (localhost, executor driver, partition 2, NODE_LOCAL, 7513 bytes)
19:02:00.138 INFO TaskSetManager - Starting task 1.0 in stage 22.0 (TID 45) (localhost, executor driver, partition 1, PROCESS_LOCAL, 7513 bytes)
19:02:00.139 INFO TaskSetManager - Starting task 3.0 in stage 22.0 (TID 46) (localhost, executor driver, partition 3, PROCESS_LOCAL, 7513 bytes)
19:02:00.139 INFO Executor - Running task 3.0 in stage 22.0 (TID 46)
19:02:00.141 INFO Executor - Running task 0.0 in stage 22.0 (TID 43)
19:02:00.144 INFO Executor - Running task 1.0 in stage 22.0 (TID 45)
19:02:00.146 INFO ShuffleBlockFetcherIterator - Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
19:02:00.146 INFO ShuffleBlockFetcherIterator - Getting 1 (54.0 B) non-empty blocks including 1 (54.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
19:02:00.146 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 2 ms
19:02:00.146 INFO ShuffleBlockFetcherIterator - Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
19:02:00.146 INFO Executor - Running task 2.0 in stage 22.0 (TID 44)
19:02:00.148 INFO ShuffleBlockFetcherIterator - Getting 1 (54.0 B) non-empty blocks including 1 (54.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
19:02:00.148 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
19:02:00.146 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
19:02:00.150 INFO Executor - Finished task 3.0 in stage 22.0 (TID 46). 1758 bytes result sent to driver
19:02:00.151 INFO TaskSetManager - Finished task 3.0 in stage 22.0 (TID 46) in 12 ms on localhost (executor driver) (1/4)
19:02:00.153 INFO Executor - Finished task 0.0 in stage 22.0 (TID 43). 1770 bytes result sent to driver
19:02:00.153 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 7 ms
19:02:00.153 INFO TaskSetManager - Finished task 0.0 in stage 22.0 (TID 43) in 15 ms on localhost (executor driver) (2/4)
19:02:00.154 INFO Executor - Finished task 1.0 in stage 22.0 (TID 45). 1758 bytes result sent to driver
19:02:00.155 INFO TaskSetManager - Finished task 1.0 in stage 22.0 (TID 45) in 17 ms on localhost (executor driver) (3/4)
19:02:00.159 INFO Executor - Finished task 2.0 in stage 22.0 (TID 44). 1770 bytes result sent to driver
19:02:00.159 INFO TaskSetManager - Finished task 2.0 in stage 22.0 (TID 44) in 21 ms on localhost (executor driver) (4/4)
19:02:00.160 INFO TaskSchedulerImpl - Removed TaskSet 22.0, whose tasks have all completed, from pool
19:02:00.160 INFO DAGScheduler - ResultStage 22 (collectAsMap at SparkSharder.java:258) finished in 0.026 s
19:02:00.160 INFO DAGScheduler - Job 16 is finished. Cancelling potential speculative or zombie tasks for this job
19:02:00.160 INFO TaskSchedulerImpl - Killing all running tasks in stage 22: Stage finished
19:02:00.161 INFO DAGScheduler - Job 16 finished: collectAsMap at SparkSharder.java:258, took 0.095810 s
19:02:00.181 INFO SparkContext - Starting job: collectAsMap at SparkSharderUnitTest.java:244
19:02:00.182 INFO DAGScheduler - Got job 17 (collectAsMap at SparkSharderUnitTest.java:244) with 4 output partitions
19:02:00.182 INFO DAGScheduler - Final stage: ResultStage 24 (collectAsMap at SparkSharderUnitTest.java:244)
19:02:00.182 INFO DAGScheduler - Parents of final stage: List(ShuffleMapStage 23)
19:02:00.183 INFO DAGScheduler - Missing parents: List()
19:02:00.183 INFO DAGScheduler - Submitting ResultStage 24 (MapPartitionsRDD[53] at flatMapToPair at SparkSharderUnitTest.java:243), which has no missing parents
19:02:00.186 INFO MemoryStore - Block broadcast_27 stored as values in memory (estimated size 10.3 KiB, free 1919.8 MiB)
19:02:00.187 INFO MemoryStore - Block broadcast_27_piece0 stored as bytes in memory (estimated size 5.1 KiB, free 1919.8 MiB)
19:02:00.187 INFO BlockManagerInfo - Added broadcast_27_piece0 in memory on localhost:45255 (size: 5.1 KiB, free: 1919.9 MiB)
19:02:00.188 INFO SparkContext - Created broadcast 27 from broadcast at DAGScheduler.scala:1580
19:02:00.188 INFO DAGScheduler - Submitting 4 missing tasks from ResultStage 24 (MapPartitionsRDD[53] at flatMapToPair at SparkSharderUnitTest.java:243) (first 15 tasks are for partitions Vector(0, 1, 2, 3))
19:02:00.188 INFO TaskSchedulerImpl - Adding task set 24.0 with 4 tasks resource profile 0
19:02:00.190 INFO TaskSetManager - Starting task 0.0 in stage 24.0 (TID 47) (localhost, executor driver, partition 0, PROCESS_LOCAL, 8821 bytes)
19:02:00.191 INFO TaskSetManager - Starting task 1.0 in stage 24.0 (TID 48) (localhost, executor driver, partition 1, PROCESS_LOCAL, 8533 bytes)
19:02:00.191 INFO TaskSetManager - Starting task 2.0 in stage 24.0 (TID 49) (localhost, executor driver, partition 2, PROCESS_LOCAL, 8677 bytes)
19:02:00.192 INFO TaskSetManager - Starting task 3.0 in stage 24.0 (TID 50) (localhost, executor driver, partition 3, PROCESS_LOCAL, 8533 bytes)
19:02:00.192 INFO Executor - Running task 3.0 in stage 24.0 (TID 50)
19:02:00.192 INFO Executor - Running task 0.0 in stage 24.0 (TID 47)
19:02:00.192 INFO Executor - Running task 2.0 in stage 24.0 (TID 49)
19:02:00.192 INFO Executor - Running task 1.0 in stage 24.0 (TID 48)
19:02:00.199 INFO BlockManager - Found block rdd_47_1 locally
19:02:00.199 INFO BlockManager - Found block rdd_47_0 locally
19:02:00.200 INFO Executor - Finished task 1.0 in stage 24.0 (TID 48). 941 bytes result sent to driver
19:02:00.200 INFO Executor - Finished task 0.0 in stage 24.0 (TID 47). 1076 bytes result sent to driver
19:02:00.200 INFO BlockManager - Found block rdd_47_2 locally
19:02:00.201 INFO BlockManager - Found block rdd_47_3 locally
19:02:00.201 INFO Executor - Finished task 2.0 in stage 24.0 (TID 49). 1012 bytes result sent to driver
19:02:00.201 INFO Executor - Finished task 3.0 in stage 24.0 (TID 50). 898 bytes result sent to driver
19:02:00.202 INFO TaskSetManager - Finished task 1.0 in stage 24.0 (TID 48) in 12 ms on localhost (executor driver) (1/4)
19:02:00.202 INFO TaskSetManager - Finished task 2.0 in stage 24.0 (TID 49) in 11 ms on localhost (executor driver) (2/4)
19:02:00.203 INFO TaskSetManager - Finished task 3.0 in stage 24.0 (TID 50) in 12 ms on localhost (executor driver) (3/4)
19:02:00.203 INFO TaskSetManager - Finished task 0.0 in stage 24.0 (TID 47) in 13 ms on localhost (executor driver) (4/4)
19:02:00.203 INFO TaskSchedulerImpl - Removed TaskSet 24.0, whose tasks have all completed, from pool
19:02:00.203 INFO DAGScheduler - ResultStage 24 (collectAsMap at SparkSharderUnitTest.java:244) finished in 0.019 s
19:02:00.204 INFO DAGScheduler - Job 17 is finished. Cancelling potential speculative or zombie tasks for this job
19:02:00.204 INFO TaskSchedulerImpl - Killing all running tasks in stage 24: Stage finished
19:02:00.204 INFO DAGScheduler - Job 17 finished: collectAsMap at SparkSharderUnitTest.java:244, took 0.022731 s
19:02:00.206 INFO MemoryStore - Block broadcast_28 stored as values in memory (estimated size 616.0 B, free 1919.8 MiB)
19:02:00.207 INFO MemoryStore - Block broadcast_28_piece0 stored as bytes in memory (estimated size 320.0 B, free 1919.8 MiB)
19:02:00.207 INFO BlockManagerInfo - Added broadcast_28_piece0 in memory on localhost:45255 (size: 320.0 B, free: 1919.9 MiB)
19:02:00.208 INFO SparkContext - Created broadcast 28 from broadcast at SparkSharder.java:100
19:02:00.236 INFO SparkContext - Starting job: collectAsMap at SparkSharderUnitTest.java:249
19:02:00.237 INFO DAGScheduler - Registering RDD 54 (flatMapToPair at SparkSharder.java:101) as input to shuffle 5
19:02:00.237 INFO DAGScheduler - Got job 18 (collectAsMap at SparkSharderUnitTest.java:249) with 4 output partitions
19:02:00.237 INFO DAGScheduler - Final stage: ResultStage 26 (collectAsMap at SparkSharderUnitTest.java:249)
19:02:00.237 INFO DAGScheduler - Parents of final stage: List(ShuffleMapStage 25)
19:02:00.237 INFO DAGScheduler - Missing parents: List(ShuffleMapStage 25)
19:02:00.238 INFO DAGScheduler - Submitting ShuffleMapStage 25 (MapPartitionsRDD[54] at flatMapToPair at SparkSharder.java:101), which has no missing parents
19:02:00.240 INFO MemoryStore - Block broadcast_29 stored as values in memory (estimated size 7.2 KiB, free 1919.8 MiB)
19:02:00.241 INFO MemoryStore - Block broadcast_29_piece0 stored as bytes in memory (estimated size 3.8 KiB, free 1919.8 MiB)
19:02:00.241 INFO BlockManagerInfo - Added broadcast_29_piece0 in memory on localhost:45255 (size: 3.8 KiB, free: 1919.9 MiB)
19:02:00.242 INFO SparkContext - Created broadcast 29 from broadcast at DAGScheduler.scala:1580
19:02:00.242 INFO DAGScheduler - Submitting 4 missing tasks from ShuffleMapStage 25 (MapPartitionsRDD[54] at flatMapToPair at SparkSharder.java:101) (first 15 tasks are for partitions Vector(0, 1, 2, 3))
19:02:00.242 INFO TaskSchedulerImpl - Adding task set 25.0 with 4 tasks resource profile 0
19:02:00.243 INFO TaskSetManager - Starting task 0.0 in stage 25.0 (TID 51) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7681 bytes)
19:02:00.244 INFO TaskSetManager - Starting task 1.0 in stage 25.0 (TID 52) (localhost, executor driver, partition 1, PROCESS_LOCAL, 7681 bytes)
19:02:00.244 INFO TaskSetManager - Starting task 2.0 in stage 25.0 (TID 53) (localhost, executor driver, partition 2, PROCESS_LOCAL, 7681 bytes)
19:02:00.244 INFO TaskSetManager - Starting task 3.0 in stage 25.0 (TID 54) (localhost, executor driver, partition 3, PROCESS_LOCAL, 7681 bytes)
19:02:00.245 INFO Executor - Running task 2.0 in stage 25.0 (TID 53)
19:02:00.245 INFO Executor - Running task 1.0 in stage 25.0 (TID 52)
19:02:00.246 INFO Executor - Running task 3.0 in stage 25.0 (TID 54)
19:02:00.248 INFO Executor - Running task 0.0 in stage 25.0 (TID 51)
19:02:00.262 INFO Executor - Finished task 1.0 in stage 25.0 (TID 52). 1065 bytes result sent to driver
19:02:00.263 INFO TaskSetManager - Finished task 1.0 in stage 25.0 (TID 52) in 20 ms on localhost (executor driver) (1/4)
19:02:00.264 INFO Executor - Finished task 2.0 in stage 25.0 (TID 53). 1065 bytes result sent to driver
19:02:00.262 INFO Executor - Finished task 3.0 in stage 25.0 (TID 54). 1065 bytes result sent to driver
19:02:00.265 INFO Executor - Finished task 0.0 in stage 25.0 (TID 51). 1065 bytes result sent to driver
19:02:00.265 INFO TaskSetManager - Finished task 2.0 in stage 25.0 (TID 53) in 21 ms on localhost (executor driver) (2/4)
19:02:00.266 INFO TaskSetManager - Finished task 3.0 in stage 25.0 (TID 54) in 22 ms on localhost (executor driver) (3/4)
19:02:00.266 INFO TaskSetManager - Finished task 0.0 in stage 25.0 (TID 51) in 23 ms on localhost (executor driver) (4/4)
19:02:00.266 INFO TaskSchedulerImpl - Removed TaskSet 25.0, whose tasks have all completed, from pool
19:02:00.267 INFO DAGScheduler - ShuffleMapStage 25 (flatMapToPair at SparkSharder.java:101) finished in 0.029 s
19:02:00.267 INFO DAGScheduler - looking for newly runnable stages
19:02:00.267 INFO DAGScheduler - running: HashSet()
19:02:00.267 INFO DAGScheduler - waiting: HashSet(ResultStage 26)
19:02:00.267 INFO DAGScheduler - failed: HashSet()
19:02:00.267 INFO DAGScheduler - Submitting ResultStage 26 (MapPartitionsRDD[58] at flatMapToPair at SparkSharderUnitTest.java:248), which has no missing parents
19:02:00.269 INFO MemoryStore - Block broadcast_30 stored as values in memory (estimated size 9.4 KiB, free 1919.8 MiB)
19:02:00.270 INFO MemoryStore - Block broadcast_30_piece0 stored as bytes in memory (estimated size 4.7 KiB, free 1919.8 MiB)
19:02:00.270 INFO BlockManagerInfo - Added broadcast_30_piece0 in memory on localhost:45255 (size: 4.7 KiB, free: 1919.9 MiB)
19:02:00.271 INFO SparkContext - Created broadcast 30 from broadcast at DAGScheduler.scala:1580
19:02:00.271 INFO DAGScheduler - Submitting 4 missing tasks from ResultStage 26 (MapPartitionsRDD[58] at flatMapToPair at SparkSharderUnitTest.java:248) (first 15 tasks are for partitions Vector(0, 1, 2, 3))
19:02:00.271 INFO TaskSchedulerImpl - Adding task set 26.0 with 4 tasks resource profile 0
19:02:00.272 INFO TaskSetManager - Starting task 1.0 in stage 26.0 (TID 55) (localhost, executor driver, partition 1, NODE_LOCAL, 7513 bytes)
19:02:00.273 INFO TaskSetManager - Starting task 0.0 in stage 26.0 (TID 56) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7513 bytes)
19:02:00.273 INFO TaskSetManager - Starting task 2.0 in stage 26.0 (TID 57) (localhost, executor driver, partition 2, PROCESS_LOCAL, 7513 bytes)
19:02:00.273 INFO TaskSetManager - Starting task 3.0 in stage 26.0 (TID 58) (localhost, executor driver, partition 3, PROCESS_LOCAL, 7513 bytes)
19:02:00.273 INFO Executor - Running task 0.0 in stage 26.0 (TID 56)
19:02:00.273 INFO Executor - Running task 1.0 in stage 26.0 (TID 55)
19:02:00.276 INFO Executor - Running task 3.0 in stage 26.0 (TID 58)
19:02:00.278 INFO Executor - Running task 2.0 in stage 26.0 (TID 57)
19:02:00.281 INFO ShuffleBlockFetcherIterator - Getting 4 (777.0 B) non-empty blocks including 4 (777.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
19:02:00.281 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
19:02:00.281 INFO ShuffleBlockFetcherIterator - Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
19:02:00.281 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
19:02:00.282 INFO Executor - Finished task 0.0 in stage 26.0 (TID 56). 1758 bytes result sent to driver
19:02:00.284 INFO ShuffleBlockFetcherIterator - Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
19:02:00.284 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
19:02:00.285 INFO Executor - Finished task 2.0 in stage 26.0 (TID 57). 1758 bytes result sent to driver
19:02:00.286 INFO ShuffleBlockFetcherIterator - Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
19:02:00.286 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
19:02:00.286 INFO Executor - Finished task 3.0 in stage 26.0 (TID 58). 1758 bytes result sent to driver
19:02:00.287 INFO TaskSetManager - Finished task 0.0 in stage 26.0 (TID 56) in 15 ms on localhost (executor driver) (1/4)
19:02:00.288 INFO TaskSetManager - Finished task 2.0 in stage 26.0 (TID 57) in 15 ms on localhost (executor driver) (2/4)
19:02:00.288 INFO TaskSetManager - Finished task 3.0 in stage 26.0 (TID 58) in 15 ms on localhost (executor driver) (3/4)
19:02:00.295 INFO Executor - Finished task 1.0 in stage 26.0 (TID 55). 1871 bytes result sent to driver
19:02:00.295 INFO TaskSetManager - Finished task 1.0 in stage 26.0 (TID 55) in 23 ms on localhost (executor driver) (4/4)
19:02:00.295 INFO TaskSchedulerImpl - Removed TaskSet 26.0, whose tasks have all completed, from pool
19:02:00.296 INFO DAGScheduler - ResultStage 26 (collectAsMap at SparkSharderUnitTest.java:249) finished in 0.028 s
19:02:00.296 INFO DAGScheduler - Job 18 is finished. Cancelling potential speculative or zombie tasks for this job
19:02:00.296 INFO TaskSchedulerImpl - Killing all running tasks in stage 26: Stage finished
19:02:00.297 INFO DAGScheduler - Job 18 finished: collectAsMap at SparkSharderUnitTest.java:249, took 0.060453 s
19:02:00.298 INFO MemoryStore - Block broadcast_31 stored as values in memory (estimated size 616.0 B, free 1919.8 MiB)
19:02:00.299 INFO MemoryStore - Block broadcast_31_piece0 stored as bytes in memory (estimated size 320.0 B, free 1919.8 MiB)
19:02:00.299 INFO BlockManagerInfo - Added broadcast_31_piece0 in memory on localhost:45255 (size: 320.0 B, free: 1919.9 MiB)
19:02:00.299 INFO SparkContext - Created broadcast 31 from broadcast at SparkSharder.java:100
19:02:00.324 INFO SparkContext - Starting job: collect at SparkSharderUnitTest.java:254
19:02:00.325 INFO DAGScheduler - Registering RDD 59 (flatMapToPair at SparkSharder.java:101) as input to shuffle 6
19:02:00.325 INFO DAGScheduler - Got job 19 (collect at SparkSharderUnitTest.java:254) with 4 output partitions
19:02:00.325 INFO DAGScheduler - Final stage: ResultStage 28 (collect at SparkSharderUnitTest.java:254)
19:02:00.325 INFO DAGScheduler - Parents of final stage: List(ShuffleMapStage 27)
19:02:00.326 INFO DAGScheduler - Missing parents: List(ShuffleMapStage 27)
19:02:00.327 INFO DAGScheduler - Submitting ShuffleMapStage 27 (MapPartitionsRDD[59] at flatMapToPair at SparkSharder.java:101), which has no missing parents
19:02:00.328 INFO MemoryStore - Block broadcast_32 stored as values in memory (estimated size 7.2 KiB, free 1919.8 MiB)
19:02:00.329 INFO MemoryStore - Block broadcast_32_piece0 stored as bytes in memory (estimated size 3.8 KiB, free 1919.7 MiB)
19:02:00.329 INFO BlockManagerInfo - Added broadcast_32_piece0 in memory on localhost:45255 (size: 3.8 KiB, free: 1919.9 MiB)
19:02:00.330 INFO SparkContext - Created broadcast 32 from broadcast at DAGScheduler.scala:1580
19:02:00.330 INFO DAGScheduler - Submitting 4 missing tasks from ShuffleMapStage 27 (MapPartitionsRDD[59] at flatMapToPair at SparkSharder.java:101) (first 15 tasks are for partitions Vector(0, 1, 2, 3))
19:02:00.330 INFO TaskSchedulerImpl - Adding task set 27.0 with 4 tasks resource profile 0
19:02:00.331 INFO TaskSetManager - Starting task 0.0 in stage 27.0 (TID 59) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7681 bytes)
19:02:00.332 INFO TaskSetManager - Starting task 1.0 in stage 27.0 (TID 60) (localhost, executor driver, partition 1, PROCESS_LOCAL, 7681 bytes)
19:02:00.332 INFO TaskSetManager - Starting task 2.0 in stage 27.0 (TID 61) (localhost, executor driver, partition 2, PROCESS_LOCAL, 7681 bytes)
19:02:00.332 INFO TaskSetManager - Starting task 3.0 in stage 27.0 (TID 62) (localhost, executor driver, partition 3, PROCESS_LOCAL, 7681 bytes)
19:02:00.332 INFO Executor - Running task 2.0 in stage 27.0 (TID 61)
19:02:00.333 INFO Executor - Running task 0.0 in stage 27.0 (TID 59)
19:02:00.333 INFO Executor - Running task 3.0 in stage 27.0 (TID 62)
19:02:00.333 INFO Executor - Running task 1.0 in stage 27.0 (TID 60)
19:02:00.349 INFO Executor - Finished task 3.0 in stage 27.0 (TID 62). 1065 bytes result sent to driver
19:02:00.350 INFO TaskSetManager - Finished task 3.0 in stage 27.0 (TID 62) in 18 ms on localhost (executor driver) (1/4)
19:02:00.352 INFO Executor - Finished task 2.0 in stage 27.0 (TID 61). 1065 bytes result sent to driver
19:02:00.353 INFO TaskSetManager - Finished task 2.0 in stage 27.0 (TID 61) in 21 ms on localhost (executor driver) (2/4)
19:02:00.355 INFO Executor - Finished task 1.0 in stage 27.0 (TID 60). 1065 bytes result sent to driver
19:02:00.355 INFO TaskSetManager - Finished task 1.0 in stage 27.0 (TID 60) in 24 ms on localhost (executor driver) (3/4)
19:02:00.357 INFO Executor - Finished task 0.0 in stage 27.0 (TID 59). 1065 bytes result sent to driver
19:02:00.358 INFO TaskSetManager - Finished task 0.0 in stage 27.0 (TID 59) in 27 ms on localhost (executor driver) (4/4)
19:02:00.358 INFO TaskSchedulerImpl - Removed TaskSet 27.0, whose tasks have all completed, from pool
19:02:00.358 INFO DAGScheduler - ShuffleMapStage 27 (flatMapToPair at SparkSharder.java:101) finished in 0.031 s
19:02:00.359 INFO DAGScheduler - looking for newly runnable stages
19:02:00.359 INFO DAGScheduler - running: HashSet()
19:02:00.359 INFO DAGScheduler - waiting: HashSet(ResultStage 28)
19:02:00.359 INFO DAGScheduler - failed: HashSet()
19:02:00.359 INFO DAGScheduler - Submitting ResultStage 28 (MapPartitionsRDD[63] at flatMapToPair at SparkSharderUnitTest.java:254), which has no missing parents
19:02:00.361 INFO MemoryStore - Block broadcast_33 stored as values in memory (estimated size 9.4 KiB, free 1919.7 MiB)
19:02:00.362 INFO MemoryStore - Block broadcast_33_piece0 stored as bytes in memory (estimated size 4.8 KiB, free 1919.7 MiB)
19:02:00.363 INFO BlockManagerInfo - Added broadcast_33_piece0 in memory on localhost:45255 (size: 4.8 KiB, free: 1919.9 MiB)
19:02:00.363 INFO SparkContext - Created broadcast 33 from broadcast at DAGScheduler.scala:1580
19:02:00.364 INFO DAGScheduler - Submitting 4 missing tasks from ResultStage 28 (MapPartitionsRDD[63] at flatMapToPair at SparkSharderUnitTest.java:254) (first 15 tasks are for partitions Vector(0, 1, 2, 3))
19:02:00.364 INFO TaskSchedulerImpl - Adding task set 28.0 with 4 tasks resource profile 0
19:02:00.365 INFO TaskSetManager - Starting task 1.0 in stage 28.0 (TID 63) (localhost, executor driver, partition 1, NODE_LOCAL, 7513 bytes)
19:02:00.365 INFO TaskSetManager - Starting task 0.0 in stage 28.0 (TID 64) (localhost, executor driver, partition 0, PROCESS_LOCAL, 7513 bytes)
19:02:00.365 INFO TaskSetManager - Starting task 2.0 in stage 28.0 (TID 65) (localhost, executor driver, partition 2, PROCESS_LOCAL, 7513 bytes)
19:02:00.366 INFO TaskSetManager - Starting task 3.0 in stage 28.0 (TID 66) (localhost, executor driver, partition 3, PROCESS_LOCAL, 7513 bytes)
19:02:00.366 INFO Executor - Running task 1.0 in stage 28.0 (TID 63)
19:02:00.367 INFO Executor - Running task 3.0 in stage 28.0 (TID 66)
19:02:00.368 INFO Executor - Running task 0.0 in stage 28.0 (TID 64)
19:02:00.370 INFO ShuffleBlockFetcherIterator - Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
19:02:00.370 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
19:02:00.371 INFO Executor - Finished task 3.0 in stage 28.0 (TID 66). 1758 bytes result sent to driver
19:02:00.372 INFO ShuffleBlockFetcherIterator - Getting 4 (777.0 B) non-empty blocks including 4 (777.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
19:02:00.372 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
19:02:00.373 INFO Executor - Running task 2.0 in stage 28.0 (TID 65)
19:02:00.375 INFO TaskSetManager - Finished task 3.0 in stage 28.0 (TID 66) in 10 ms on localhost (executor driver) (1/4)
19:02:00.375 INFO ShuffleBlockFetcherIterator - Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
19:02:00.375 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
19:02:00.376 INFO ShuffleBlockFetcherIterator - Getting 0 (0.0 B) non-empty blocks including 0 (0.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) push-merged-local and 0 (0.0 B) remote blocks
19:02:00.376 INFO ShuffleBlockFetcherIterator - Started 0 remote fetches in 0 ms
19:02:00.377 INFO Executor - Finished task 0.0 in stage 28.0 (TID 64). 1758 bytes result sent to driver
19:02:00.378 INFO TaskSetManager - Finished task 0.0 in stage 28.0 (TID 64) in 13 ms on localhost (executor driver) (2/4)
19:02:00.379 INFO Executor - Finished task 2.0 in stage 28.0 (TID 65). 1758 bytes result sent to driver
19:02:00.381 INFO TaskSetManager - Finished task 2.0 in stage 28.0 (TID 65) in 16 ms on localhost (executor driver) (3/4)
19:02:00.383 INFO Executor - Finished task 1.0 in stage 28.0 (TID 63). 1871 bytes result sent to driver
19:02:00.384 INFO TaskSetManager - Finished task 1.0 in stage 28.0 (TID 63) in 20 ms on localhost (executor driver) (4/4)
19:02:00.384 INFO TaskSchedulerImpl - Removed TaskSet 28.0, whose tasks have all completed, from pool
19:02:00.384 INFO DAGScheduler - ResultStage 28 (collect at SparkSharderUnitTest.java:254) finished in 0.024 s
19:02:00.384 INFO DAGScheduler - Job 19 is finished. Cancelling potential speculative or zombie tasks for this job
19:02:00.384 INFO TaskSchedulerImpl - Killing all running tasks in stage 28: Stage finished
19:02:00.385 INFO DAGScheduler - Job 19 finished: collect at SparkSharderUnitTest.java:254, took 0.060201 s