Featured Job

Senior Staff Engineer, Data Platform

United States Full-time Hybrid $215k — $235k per year 11/20/2026 Job ID: 000627
Apply Now
Databricks Lakehouse architecture Medallion layering Delta Lake Unity Catalog

Summary

What you’ll impact

The Lead Data Platform Engineer - Healthcare defines and builds a scalable lakehouse platform that powers AI and analytics. The team uses a unified data analytics platform, a big data processing framework, an open-source storage layer, a data catalog, Python, SQL, and a workflow orchestration tool within cloud environments (major cloud providers) and infrastructure-as-code tools. Responsibilities include architecting data pipelines, enforcing PHI‑aware governance, creating LLM‑ready datasets, and designing durable workflow orchestrations. The role partners with product, AI, and engineering teams, sets platform standards, and mentors engineers across multiple squads.

Responsibilities

What you'll do

  • Own the lakehouse. Architect and evolve our Databricks lakehouse, including medallion layering (bronze, silver, gold), Delta Lake, Unity Catalog governance, and workspace and compute strategy.
  • Build and tune Databricks pipelines. Design, build, and optimize Spark and SQL pipelines on Databricks, including incremental processing, data quality enforcement, performance tuning, and cost management.
  • Make data trustworthy. Define data contracts, quality checks, access controls, and PHI-aware governance in partnership with security and compliance.
  • Power AI and LLM workloads. Build the data foundations for AI products: LLM-ready datasets, retrieval and vector-store pipelines, evaluation data, and the interfaces that let AI and product teams consume governed data safely.
  • Build AI pipelines on Temporal. In addition to Databricks, design durable workflows on Temporal for AI pipelines: scheduled and watermarked pulls from Databricks gold tables, file-ingestion workflows, and long-running LLM extraction and enrichment jobs. Establish patterns for retries, idempotency, versioning, and backfills, and keep the handoff between Databricks and Temporal clean, observable, and well governed.
  • Lead agentic engineering. Use AI coding agents daily and define how the team does agentic development: tooling, prompts, guardrails, code review practices, and test strategy for agent-written code.
  • Set technical direction. Write and review technical designs, set platform-wide standards, and make build-versus-buy decisions that hold up over multiple years.
  • Multiply the team. Mentor engineers across teams, unblock other pods, and communicate trade-offs clearly to technical and non-technical stakeholders.

Requirements

What you’ll bring

  • Bachelors Degree in Computer Science, Electrical Engineering, Mathematics or a related field
  • At least 12+ years of software or data engineering experience, with a track record of designing and operating production data platforms at scale and of setting architecture across multiple teams.
  • Experience working in a highly regulated domain (Healthcare, Insurance, Finance) is required
  • Deep, hands-on expertise with Databricks is highly preferred but expertise in Snowflake or Microsoft Fabric would be aligned as well.
  • Experience with the lakehouse stack: Spark, Delta Lake, Unity Catalog, Workflows or equivalent, and performance and cost tuning at scale.
  • Experience building pipelines for ML or LLM applications, such as LLM-based extraction, embeddings and vector search, feature pipelines, or evaluation datasets.
  • Fluency in Python and SQL, with sound software engineering practices: testing, CI/CD, infrastructure as code, and code review.
  • Demonstrated practical use of agentic coding tools (such as Claude Code or similar) in real delivery work, and good judgment about when to trust, verify, and constrain them.
  • Experience owning ambiguous, cross-team problems end to end and influencing without direct authority.
  • Clear written and verbal communication, including authoring design docs that others build from.
  • Experience with healthcare data: claims, EHR, FHIR or HL7, and HIPAA-compliant handling of PHI.
  • Databricks certifications or experience with Delta Live Tables, Lakeflow, or Databricks Asset Bundles.
  • Experience with other orchestrators (Airflow, Dagster, Prefect) and a view on when each is the right tool.
  • Experience with cloud infrastructure (GCP, AWS, or Azure), Terraform, and Kubernetes.
  • Contributions to open-source data, workflow, or AI infrastructure projects.

Ready to Move Forward?

Apply now and our recruiting team will reach out with next steps, interview guidance, and client insights tailored to this role.