Data Engineer (Python)

<h2>Data Engineer (Python)</h2> <h3>Company</h3> <p>Orcrist builds the Orcrist Intelligence Platform (OIP), a Kubernetes-based data intelligence system delivered as SaaS or self-hosted/on-prem (including air-gapped deployments). We run streaming and batch pipelines that power search, ML enrichment, and investigative workflows for mission-critical customers.</p> <h3>Role</h3> <p>Rapidly validate new data initiatives end-to-end—without sacrificing adoptability. On Innovation, you’ll prototype representative connectors and pipelines (batch + streaming), generate credible performance/operability readouts, and ship a handoff package that Foundation or a delivery team can productize.</p> <h3>What you'll do</h3> <ul> <li>Prototype ingestion and connector patterns (batch + streaming) using NiFi, Kafka, Kafka Connect/Streams, and CDC approaches.</li> <li>Design “prototype-grade but adoptable” schemas and data models with clear semantics and evolution discipline.</li> <li>Build incremental lakehouse datasets (Hudi/Iceberg/Delta patterns) and produce queryable outputs for realistic latency/throughput evaluation.</li> <li>Bake in data quality and provenance mindset early (checks, metadata hooks, operability basics).</li> <li>Containerize and deploy prototypes on Kubernetes; deliver minimal runbooks/configs that make adoption straightforward.</li> <li>Produce adoption artifacts: schemas, reference implementations, technical design notes, and an integration backlog.</li> </ul> <h3>About You</h3> <ul> <li>3+ years data engineering experience (level dependent) with real pipeline delivery beyond ad-hoc scripts.</li> <li>Strong Python + SQL; comfortable building transformations, validation tooling, and pipeline glue code.</li> <li>Practical streaming/CDC fundamentals (ordering, duplication, replay, idempotency) and Kafka ecosystem experience.</li> <li>Familiar with lakehouse/storage and query layers (e.g., Hudi/Iceberg/Delta, Trino/Hive/Postgres) and how to make datasets usable.</li> <li>Comfortable working in Kubernetes/container environments and documenting decisions clearly.</li> <li>Eligible to work in Germany; EU/NATO citizenship preferred and export-control screening applies.</li> </ul> <h3>Nice‑to‑haves</h3> <ul> <li>Great Expectations or similar data quality tooling; metadata/lineage platforms (OpenMetadata/DataHub/Atlas).</li> <li>Experience shipping in on-prem or air-gapped environments; governance/policy awareness for regulated customers.</li> <li>German language (B1+) and/or experience with OSINT/GEOINT/multi-INT data shapes.</li> </ul> <h3>What We Offer</h3> <ul> <li>Modern data stack with real constraints: Kafka + NiFi + lakehouse + distributed SQL + Kubernetes.</li> <li>Remote-first in Germany with regular Berlin prototyping sprints, 30 days vacation, equipment &amp; learning budget.</li> <li>High leverage: your prototypes become blueprints multiple teams reuse and productize.</li> </ul>