Summary
What you’ll impact
The Staff Machine Learning Engineer is responsible for designing and building the core harness that powers AI agents across the platform. The role leverages Python, large language models, distributed systems, and simulation environments to create trajectory‑level evaluation systems and action guardrails. Key deliverables include developing evaluation pipelines, building permission models, sandboxing mechanisms, and SDKs that enable other teams to adopt safety and reliability primitives. The engineer collaborates with product, infrastructure, and ML teams, mentors peers, and sets technical standards for agent deployment and governance.
Responsibilities
What you'll do
- Design and build the core of Robinhood's agent harness — orchestration, tool integrations, context and memory management — so one platform can safely power both high-trust internal agents and tightly scoped customer-facing ones.
- Ship agentic applications end to end on that harness, from an ambiguous problem to a production agent that takes real action on behalf of employees or customers, and feed what you learn back into the platform.
- Build trajectory-level evaluation systems that score how an agent got to an answer, not just the answer — tool-call correctness, planning and recovery, multi-step task completion — backed by simulation environments and synthetic task generation.
- Architect action guardrails as platform primitives: least-privilege tool scoping, permission models, human-approval gates for high-risk or irreversible actions, step and budget limits, sandboxing, and rollback.
- Make evals and guardrails products other teams adopt — SDKs, CI regression gates on prompt, model, and tool changes, continuous red-teaming, and production tracing that closes the loop from real traffic back into eval sets and guardrail models.
- Set the technical bar through architecture reviews, code reviews, and mentorship, and be the person who can make — and defend with data — the "don't ship" call.
Requirements
What you’ll bring
- 10+ years of experience as a Machine Learning Engineer or ML-focused software engineer, with strong Python and distributed-systems fundamentals and a track record of shipping LLM-powered systems to production at scale. A Master's degree in Computer Science or a related technical field, or equivalent professional experience.
- Hands-on experience building agentic systems end to end — tool use, orchestration, context management, multi-step planning — on top of frontier models, in production.
- Deep expertise evaluating agents: you've built trajectory-level evals, tool-call scoring, and simulation environments, and you can articulate why final-answer accuracy is insufficient for systems that act.
- Demonstrated expertise designing action-level guardrails — permission and tool-scoping models, approval gates, blast-radius controls, and sandboxing — for agents operating in systems where mistakes have consequences.
- Rigor in evaluation methodology: golden datasets, rubric and LLM-as-judge grading and their failure modes, statistical significance with small N, offline-to-online metric correlation, and eval data versioning and contamination control.
- Proven ability to build platforms, not just models: you've shipped eval, safety, or agent tooling that other engineering teams adopted, and you have the judgment to know when to build versus buy.