Shuffle Operation

 Shuffle operation is the most expensive operation in Spark as takes a lot of I/O, networks calls. 

Why we need shuffle operation: 

While running some RDD transformations like groupByKey, Join the data needs to be re-partition. 

For example:

groupByKey expects data of the same key on a single worker node but some of the data might be on other worker nodes, in such cases, data of the same key from all the worker nodes have to move to a single node. This shuffling of data involves transferring data on a network.

import org.apache.spark._
object Shuffle {
  def main(args: Array[String]): Unit = {
    val sc = new SparkContext("local[*]", "shuffleOperation")
    val input = List(("Scala", 123), ("R", 100), ("Scala", 199), ("Java", 555), ("Java", 77))
    val rdd = sc.parallelize(input)
    val mapInput =
      rdd.groupByKey()
        .map(x => (x._1, (x._2.size, x._2.sum)))
        .collect()
    mapInput.foreach(println)
    // Just to keep main function running so that we can check spark server at 4200.
    scala.io.StdIn.readLine()
  }
}

DAG for above program

Total time taken by this operation is 4 sec.

In this case all the data of same key moves to respective worked node and then it invokes group by key operations.

We can improve this by using redeuceByKey:

import org.apache.spark._
object ShuffleUsingReduce {
  def main(args: Array[String]): Unit = {
    val sc = new SparkContext("local[*]", "shuffleOperation")
    val input = List(("Scala", 123), ("R", 100), ("Scala", 199), ("Java", 555), ("Java", 77))
    val rdd = sc.parallelize(input)
    val mapInput =
      rdd.map(x=> (x._1, (1, x._2)))
        .reduceByKey((x,y) => (x._1 + y._1, x._2 + y._2))
        .collect()
    mapInput.foreach(println)
    // Just to keep main function running so that we can check spark server at 4200.
    scala.io.StdIn.readLine()
  }
}

Total time taken by this operation is 3 sec.

Before transferring the same key data to the respective worker node, reducebykey invokes reduce operation, so the total amount of transferred data is less than groupbykey.

In this case, only distinct/unique key data will be transferred from each node to the respective node. 

Spark

What is Spark: Spark is an open-source engine for large-scale data processing. Spark distributes data on multiple nodes machines and tries to execute them in parallel. Spark distribute tasks on different stages and further distribute stages into two parts. A stage that can be executed parallelly and a stage that needs to execute in sequential. Spark is faster than Hadoop as it can cache the data and able to execute the operations in memory.

Spark Architecture:

Driver program: The driver program is used to create spark context. SparkContext is used to create a connection with the Spark cluster and can be used further to create RDDs, accumulators, and broadcast variables on that cluster.

Cluster manager: The driver program and SparkContext work with the cluster manager to split the task after the creation of RDD. The task is distributed into nodes to execute them parallel. Cluster Manager can be Hadoop yarn, apache mesos, or spark cluster. 

Worker node: The worker node has an executor which runs the task/program. A worker node is a machine from the Spark cluster. 

Spark can create a parallel collection of data or it can parallelize the data using RDD. We can split the RDD into different blocks. These blocks can be executed parallel onto the worker node.

Spark figures out the task execution path using DAG. In DAG, The node or worker machine is referred to as partition, and operation is referred to as edge.