Apache Spark

Also known as: Spark

On this page 7

Apache Spark is an open-source, distributed computing engine for large-scale data processing that splits work across a cluster of machines and keeps intermediate results in memory wherever possible. In the field of Data, Spark is the dominant general-purpose framework for batch processing, SQL analytics, stream processing and machine learning on datasets that are too large for a single computer.

Spark was created as a faster, more expressive successor to MapReduce. Instead of writing every intermediate step to disk, it builds a directed acyclic graph (DAG) of operations and executes it in optimized stages, which typically makes iterative and interactive workloads several times faster than the disk-bound model it replaced.

How Apache Spark executes a job

A Spark application consists of one driver program and many executors. The driver holds the user code, plans the computation and schedules tasks, while executors are worker processes that run those tasks on partitions of the data and cache results. A cluster manager such as Kubernetes, YARN or the built-in standalone manager allocates the machines.

Spark uses lazy evaluation: transformations such as filter, select or join only describe the computation. Nothing runs until an action such as count, collect or write requests a result. At that point the driver turns the logical plan into a physical plan, splits it into stages at shuffle boundaries, and sends one task per partition to the executors.

The following table summarizes the main runtime components and the role each one plays.

ComponentRoleTypical failure symptom
DriverPlans the job, tracks tasks, collects resultsOut-of-memory error after a large collect
ExecutorRuns tasks and caches partitionsLost executor, task retries
Cluster managerAllocates CPU and memory to executorsPending jobs waiting for resources
TaskProcesses one partition in one stageStraggler task delaying the stage
ShuffleRedistributes data between stages by keyHeavy disk spill and network traffic
Core components of a Spark application

Fault tolerance comes from lineage rather than replication. Every dataset remembers the transformations that produced it, so when an executor fails, Spark recomputes only the lost partitions from their sources instead of restarting the whole job.

Spark APIs and libraries

Spark exposes several abstraction levels on top of the same engine. The original resilient distributed dataset (RDD) offers low-level control, while DataFrames and Datasets add a schema and let the Catalyst optimizer rewrite queries, push filters down to storage and choose join strategies. The Tungsten execution layer then generates compact bytecode and manages memory off the Java heap.

The table below compares the main interfaces and the libraries built on them.

Interface or libraryPurposeBest suited for
RDD APILow-level functional operationsUnstructured data, custom partitioning
DataFrame APITabular operations with a schemaMost data transformation pipelines
Spark SQLANSI SQL over tables and filesAnalysts and reporting queries
Structured StreamingIncremental queries on unbounded dataNear-real-time pipelines
MLlibDistributed machine learning algorithmsFeature engineering and model training at scale
Main Spark interfaces and when they fit

Code can be written in Scala, Java, Python or R. The Python interface, PySpark, is the most widely used today; its performance is close to Scala when code stays within DataFrame operations, but Python user-defined functions can be much slower because rows must cross between the Java and Python processes.

Spark sits between classic disk-based batch engines and dedicated streaming engines. It generalized the map and reduce pattern into arbitrary DAGs, and its streaming engine processes data in small incremental units rather than one event at a time, which shapes its latency profile.

ApproachExecution modelTypical latencyMain strength
Apache SparkIn-memory DAG of stagesSeconds to minutesOne engine for batch, SQL and ML
MapReduceMap and reduce with disk between stepsMinutes to hoursSimplicity and robustness
Apache FlinkContinuous event-at-a-time dataflowMillisecondsLow-latency stateful streaming
Single-node dataframesIn-process on one machineSubsecond to minutesNo cluster overhead for small data
Cloud data warehouseManaged SQL engine on columnar storageSecondsLittle infrastructure to operate
Spark versus neighboring approaches

A common misconception is that Spark is always faster than alternatives. For datasets that fit comfortably on one machine, a single-node tool often finishes sooner because it avoids scheduling, serialization and network costs. Spark pays off when data volume, parallelism or the need for horizontal scaling justifies a cluster.

Tuning Spark for performance

Most Spark performance work concerns partitioning, shuffles and memory. A good partition usually holds roughly 100 to 200 MB of data, enough to keep tasks efficient without exhausting executor memory. Too few partitions leave cores idle; too many create scheduling overhead and thousands of tiny output files.

The following decision table links frequent situations to an appropriate configuration choice.

SituationRecommended choiceReason
Joining a large table with a small oneBroadcast joinAvoids shuffling the large table
One key holds most of the rowsEnable skew handling or salt keysPrevents a single straggler task
Same dataset reused in several actionsCache or persist itAvoids recomputation from lineage
Uneven partition sizes after filteringRely on adaptive query executionCoalesces and splits partitions at runtime
Many small output filesRepartition before writingImproves downstream read performance
Common tuning decisions

Since Spark 3, adaptive query execution (AQE) re-optimizes plans using statistics gathered during execution, which removes much of the manual tuning that earlier versions required. Columnar formats such as Parquet combined with partition pruning and predicate pushdown often matter more than any engine setting.

Limitations and common mistakes

Spark is powerful but unforgiving of careless patterns. The most expensive errors usually come from moving too much data to the driver or across the network.

  • Calling collect on large data pulls every row into the driver and frequently crashes it.
  • Unnecessary shuffles from repeated groupBy or distinct operations dominate run time.
  • Row-by-row Python UDFs bypass the optimizer; built-in functions are usually far faster.
  • Caching everything wastes executor memory and can force spills that slow the job.
  • Streaming latency below about 100 ms is not a realistic target for the default micro-batch mode.

Operational cost is another limitation. Clusters must be sized, monitored and upgraded, which is why many teams run Spark through managed platforms that handle provisioning and autoscaling.

A non-technical view

Imagine counting every word in a library with millions of books. One person would need years. Apache Spark works like a well-organized team: a coordinator splits the books into piles, hands one pile to each helper, and then combines their tallies into one final answer.

What makes Spark quick is that the helpers keep their notes on the desk instead of filing them away after every step. If one helper goes home sick, the coordinator knows exactly which pile was unfinished and gives it to someone else. The key idea is simple: split big data work into many small pieces and run them at the same time.

A concrete example

An online retailer collects about 2 TB of clickstream events per day as Parquet files in object storage. Every night, a Spark job on a cluster of 40 executors reads the previous day, removes bot traffic, deduplicates sessions, joins the events with a small product catalog through a broadcast join, and aggregates revenue and conversion rates by category and region.

Before tuning, the job took almost three hours because one popular product dominated a join key and a single task processed a third of the data. After the team enabled adaptive skew handling, raised the partition count and replaced two Python functions with built-in expressions, the same pipeline finished in about 35 minutes, leaving time for the analytics team to refresh dashboards before the working day began.