Skip to content

Lakehouse: overview

Six catalog templates together form a lakehouse on the platform's S3 buckets: an Apache Iceberg table catalog (implemented by Apache Polaris), the distributed SQL engine Trino, JupyterHub notebooks, a Spark Connect server, an MLflow tracking server and Apache NiFi for ingestion. Deploy the catalog first, then the rest in any order, in the same namespace.

Lakehouse catalog (Processing, Governance, Data Science categories)

flowchart LR
    B[(S3 bucket<br/>warehouse)]
    P[Apache Iceberg<br/>Polaris catalog]
    T[Trino]
    J[JupyterHub]
    S[Spark Connect]
    M[MLflow]
    N[Apache NiFi]
    T -- metadata --> P
    J -- metadata --> P
    P -- metadata files --> B
    T -- Parquet data --> B
    J -- data --> B
    J -- Spark Connect --> S
    S -- metadata --> P
    S -- data --> B
    J -- runs --> M
    M -- artifacts --> B
    N -- ingestion --> B

Applications