spark-submit and Deployment

Analogy categoryAnalogies

spark-submit is the standard command-line launcher for Spark applications, whether written in Scala, Java, Python, or R, and it works identically across standalone, YARN, and Kubernetes cluster managers by accepting a common set of flags such as --master, --deploy-mode, --class, --conf, and application arguments. It resolves and ships application code plus dependencies to the cluster, then hands control to the specified cluster manager to actually launch the driver and executors.

🏏

Cricket analogy: The team manager filling out a standard match-day team sheet before every game, listing playing XI and roles, is like spark-submit's common flag format that works the same whether the match is a Test, ODI, or T20.

Client Mode vs. Cluster Mode

In client deploy mode, the driver runs on the machine that invokes spark-submit — useful for interactive work like spark-shell or notebooks because you see logs immediately and can react to output — but the driver's process must stay alive and network-reachable by executors for the whole job. In cluster deploy mode, spark-submit hands the driver off to run inside the cluster itself (as a YARN ApplicationMaster container or a Kubernetes pod), which is preferred for production jobs because the driver isn't tied to your terminal and survives you disconnecting or your laptop going to sleep.

🏏

Cricket analogy: A captain calling shots live from the boundary rope during a club match, needing to stay present the whole time, is like client mode, whereas a captain who delegates tactics to a vice-captain fully embedded in the middle order resembles cluster mode's independence.

bash

# Client mode: driver stays on this machine, good for spark-shell / notebooks
spark-submit \
  --master yarn \
  --deploy-mode client \
  --class com.example.DailyReport \
  report-job.jar

# Cluster mode: driver runs inside the cluster, safe to close your laptop
spark-submit \
  --master yarn \
  --deploy-mode cluster \
  --class com.example.DailyReport \
  report-job.jar

Packaging and Dependencies

Beyond the application's own script or JAR, real Spark jobs usually need extra dependencies: --jars ships additional JAR files to the classpath, --py-files ships zipped Python modules or egg/wheel files alongside a PySpark job, and --packages resolves Maven coordinates like groupId:artifactId:version straight from a repository at submit time, which is convenient for connectors like spark-sql-kafka. For production, many teams instead build a single uber/fat JAR (or a Python wheel plus a requirements-pinned virtual environment) so the exact dependency versions are locked and reproducible rather than resolved fresh on every submit.

🏏

Cricket analogy: A team travelling for an overseas tour packs its own specialized kit like specific spikes for English pitches rather than relying on whatever's available locally, similar to using --jars to ship exact dependency versions instead of hoping the cluster has them.

In client mode, the driver's network connection to every executor must stay open for the entire job. If you run spark-submit --deploy-mode client from a laptop and it loses network connectivity or goes to sleep, executors lose contact with the driver and the whole application fails — always use cluster mode for unattended production jobs.

Configuration Precedence

Spark resolves configuration from multiple sources with a clear precedence order: values set programmatically in code via SparkConf or SparkSession.builder().config() win, then values passed as --conf flags or dedicated flags like --executor-memory on the spark-submit command line, and finally defaults read from the spark-defaults.conf file on the submitting machine, which acts as a cluster-wide fallback for anything not explicitly overridden. This layering lets platform teams set safe organization-wide defaults in spark-defaults.conf while individual jobs override just the settings they need at submit time or in code.

🏏

Cricket analogy: A player's personal grip preference set in the dressing room overrides the team's standard bat, which itself overrides the manufacturer's factory default, mirroring how code-level config overrides spark-submit flags which override spark-defaults.conf.

--packages resolves dependencies from Maven Central (or a configured repository) at submit time over the network, which is convenient for prototyping but adds startup latency and a runtime dependency on repository availability; pinning exact versions in a pre-built uber JAR avoids both problems for production.

spark-submit is the uniform launcher for Spark applications across standalone, YARN, and Kubernetes.
Client mode runs the driver on the submitting machine; cluster mode runs the driver inside the cluster itself.
Cluster mode is preferred for production because the job doesn't depend on the submitting machine staying connected.
--jars, --py-files, and --packages ship additional dependencies; uber JARs or packed virtual environments give reproducible, pinned versions.
Spark resolves configuration with precedence: code (SparkConf) > spark-submit flags > spark-defaults.conf.
spark-defaults.conf provides cluster-wide safe defaults that individual jobs can override at submit time or in code.