Aller au contenu

Example: datawarehouse pipeline

This example walks through the standard workflow for building an end-to-end data pipeline on the platform: a Git repository to version the Apache Hop project, a PostgreSQL database as the target, Apache Hop GUI to design the pipeline, an Apache Hop Server to run it, and a recurring schedule to automate it.

Overview

  1. Create a Git repository.
  2. Create a database.
  3. Launch Apache Hop GUI, with the repository as a parameter.
  4. Create a pipeline in Hop GUI.
  5. Launch an Apache Hop Server on the same repository and branch.
  6. Create a recurring job on that pipeline.
  7. Monitor the result.

1. Create a Git repository

From the Git page, My Git account section, Create a repository — personal or in an organization. This repository will version the Apache Hop project (.hpl pipelines, .hwf workflows, connections and metadata).

2. Create a database

From the catalog (New app), deploy the PostgreSQL template — see Deploying an application. This is the target database the pipeline will write to. Note the application name: it identifies the PostgreSQL connection automatically exposed to Apache Hop (see next step).

3. Launch Hop GUI with the repository as a parameter

Deploy the Apache Hop (GUI) template. In the form's Git source section:

  • Forgejo instance: the platform-managed instance;
  • Repository: the repository created in step 1;
  • Branch: leave empty for the default branch.

Once the application is Running, open it (Open): the Hop project is synced to this repository/branch, and connections to the namespace's PostgreSQL databases, Hop servers and S3 buckets are automatically provisioned in the project's metadata.

Tip

If the database created in step 2 doesn't show up right away as a connection in Hop, use Resync Hop connections on the application's card (see Applications) instead of waiting for the periodic sync.

4. Create a pipeline

In Hop GUI, create a new pipeline (.hpl) using the PostgreSQL connection provisioned in the previous step as its target, then commit it to the repository (Hop GUI has a built-in Git client). This file is what the Hop Server will run in the following steps.

5. Launch a Hop Server on the same repository/branch

Deploy the Apache Hop Server template, with the same Git source configuration as in step 3 (same Forgejo instance, same repository, same branch) — the Hop Server must see exactly the same project, pipeline included, as the one used in Hop GUI.

6. Create a recurring job on the pipeline

Open Hop Jobs from the Hop Server's card, then, under Scheduled recurring jobs, create a schedule pointing at the pipeline created in step 4 (interval or cron expression) — see Apache Hop Server: jobs and scheduling for the field details.

7. Monitor the result

From the same Hop Jobs page:

  • the schedule's Last run column confirms it's firing as expected;
  • the history lists every run with its status (Completed, Failed…) — click a row for the pipeline's graphical detail and full logs;
  • the banner at the top of the page gives an instant view of running/queued jobs and the instance's load.