Summary
What you’ll impact
The Platform Engineer will own and operate the cloud analytics environment for government enterprise analytics, overseeing governance, security, networking, cost, and reliability. They will serve as the senior technical authority, providing automation, incident remediation, and guidance to data scientists, engineers, and analysts.
Responsibilities
What you'll do
- Own Azure Databricks account and workspace administration — configuration, governance, security posture and operational reliability.
- Design and enforce the Unity Catalog governance architecture: catalogs, schemas, external locations, storage credentials, lineage, and fine-grained access control.
- Automate workspace provisioning and configuration with Infrastructure as Code (Terraform Databricks provider) and Azure DevOps or GitHub Actions CI/CD.
- Own identity and access integration with Microsoft Entra ID — SCIM provisioning, group entitlements, service principals, managed identities.
- Design and maintain secure Azure networking: VNet injection, Private Link, secure cluster connectivity, storage firewall rules, ADLS Gen2.
- Set cluster policies, pool strategy and job compute standards; run continuous cost and performance analysis and act on it.
- Build production pipelines with Delta Lake, Lakeflow Declarative Pipelines / Delta Live Tables and Lakeflow Jobs.
- Administer Databricks SQL warehouses, queries and AI/BI dashboards for the analyst community.
- Evaluate and integrate Databricks AI capabilities where they earn their place — Genie / AI-BI, Mosaic AI, Model Serving, MCP integrations against governed data.
- Define standards, runbooks and reference architectures, and mentor engineers across the program.
- Lead root-cause analysis and permanent remediation on platform incidents.
- Partner with security staff on FISMA / NIST 800-53 control implementation and audit evidence.
Requirements
What you’ll bring
- 8+ years in data platform, data engineering or cloud infrastructure, including 5+ years hands-on Databricks account and workspace administration.
- Deep Azure Databricks experience. This platform runs on Azure; Azure-native depth is a requirement, not a preference.
- Expert Unity Catalog knowledge — governance architecture, metastore administration, external locations, lineage, fine-grained access control.
- Strong Azure platform skills: ADLS Gen2, Microsoft Entra ID, Azure networking, Key Vault, Azure DevOps.
- A track record of optimizing Databricks for cost, reliability and performance — with specific examples of spend you reduced and how.
- Infrastructure as Code for Databricks (Terraform provider or equivalent) plus CI/CD automation.
- Production experience with Delta Lake, Lakeflow Declarative Pipelines / DLT and Lakeflow Jobs.
- Strong Python and SQL; PySpark and distributed-computing fundamentals including partitioning, shuffle behavior and tuning.
- Clear communication with both engineers and non-technical stakeholders.
- Currently based in the Atlanta, GA metro area and able to work on-site.
- Ability to obtain and maintain a U.S. federal Public Trust / Suitability determination.
- Bachelor's degree.