Spark Connect¶
Part of the lakehouse (overview and interaction diagram).
The Spark Connect template (Processing category) deploys an Apache Spark server in local mode (executors inside the pod) exposed through the Spark Connect protocol: notebooks and PySpark clients of the namespace share a single Spark installation, already configured on the chosen Apache Iceberg catalog. No public route (gRPC protocol, namespace-internal).
| Field | Role |
|---|---|
| Apache Iceberg (Polaris) catalog (required) | Instance of the same namespace. Fixed at creation. |
| Iceberg catalog (warehouse) | Catalog name on the Polaris side, lakehouse by default. |
| Spark version | Tag of the ds/spark-connect image. Empty = the chart's version. |
| Warehouse S3 bucket (Storage tab) | The same bucket as the catalog's: select it to fill in endpoint, region and keys. |
| S3 endpoint, S3 region, Access Key ID, Secret Access Key (Storage tab) | Filled in by the bucket picker, editable afterwards — type them in for an external S3. |
| Spark cores (local[N]) (Capacity tab) | Number of execution threads in the pod, 2 by default; the pod's CPU limit follows. |
| Spark (driver) memory, Pod memory limit (Capacity tab) | 2g and 3Gi by default: keep ~1 Gi of headroom between Spark memory and the pod limit. |
| Require a connection token | Shared token (spark.connect.authenticate.token), kept in the <name>-platform secret. Off by default. |
From an attached JupyterHub notebook (SPARK_REMOTE pre-filled) or any pyspark-client of the same minor version as the server:
from pyspark.sql import SparkSession
spark = SparkSession.builder.remote(os.environ["SPARK_REMOTE"]).getOrCreate()
spark.sql("CREATE TABLE lakehouse.demo.t (x INT) USING iceberg")
spark.sql("SELECT * FROM lakehouse.demo.t").show()
Deletion¶
Deleting a Spark Connect instance touches nothing else; Spark jobs using it as their profile can no longer be relaunched as-is.