Senior/Staff Backend Engineer - Distributed System
Palo Alto, CAFull-timePosted 1w agoStill listed 2 days ago
Most applications go out cold — see where you stand first. No sign-up to start.
You apply on the employer's site, with no Olive account needed.
Watch jobs like this. New roles like this one near Palo Alto, CA, by email.
Don't just apply. Show up ready.
Olive works from this exact posting.
At a glance
Olive lists jobs from US employers, including remote roles you can work from the United States.
Job overview
The Backend Engineer will build systems that orchestrate GPU clusters for AI workloads, including APIs for GPU allocation, memory, scheduling, and multi-tenant isolation. The role focuses on efficient resource sharing, service quality, monitoring, and cost-conscious infrastructure. The team works in a hybrid arrangement, with three days in the office and two days working from home, and requires applicants to be located in Palo Alto.
Skills & qualifications
Skills
Qualifications
Full job description
About The Company Our hiring partner is on a mission to make AI compute ubiquitous, seamless, and limitless. They’re building a cloud where AI just works—anywhere, anytime. “AI Power. Everywhere.” Join the team designing the infrastructure for an AI-first world.
Why this role exists Our hiring partner is looking for a Backend Engineer to build the systems that orchestrate GPU clusters for AI workloads. You’ll develop APIs that manage GPU allocation, memory, compute scheduling, and multi-tenant isolation—challenges unique to AI infrastructure that go far beyond standard backend engineering. On their backend team, you’ll tackle questions like: How can high-cost GPU resources be efficiently shared among users? How do we handle memory constraints for large AI models? How do we maintain quality of service when workloads compete for compute? This is an opportunity to build infrastructure where every API call could allocate thousands of dollars of compute per hour, and where your optimizations directly influence whether AI startups can train their models affordably.
What you’ll do
- Design APIs that simplify complex GPU operations for developers
- Build scheduling algorithms that maximize GPU utilization while meeting SLAs
- Develop systems to manage the full GPU lifecycle: provisioning, allocation, scheduling, and release
- Implement usage tracking and billing for GPU-hours, memory, and compute utilization
- Create monitoring solutions for GPU-specific metrics, health checks, and automated recovery
- Build multi-tenant systems with resource isolation, quota management, and fair scheduling
- Optimize cold starts for model serving and efficient model loading
- Collaborate with frontend engineers to expose complex infrastructure through intuitive interfaces
- Leverage AI-assisted coding tools (GitHub Copilot, Claude Code, Cursor IDE, etc.) to enhance productivity and code quality
You’ll thrive here if you
- Have 5+ years of backend engineering experience in distributed systems
- Are proficient in Go, Python, or similar backend languages
- Have experience with resource scheduling, orchestration, and API design (REST, GraphQL, gRPC)
- Understand hardware constraints and system optimization
- Have Linux systems and containerization experience (Docker, Kubernetes)
- Are comfortable working with expensive resources where efficiency impacts costs
- Are excited to solve novel problems in AI infrastructure (beyond CRUD apps)
- Bring a startup mindset—comfortable with ambiguity and rapid iteration
Bonus qualifications
- Experience with GPU or HPC cluster management
- Familiarity with ML/AI workload patterns and requirements
- Experience with high-value resource allocation systems
- Background in performance optimization for compute-intensive workloads
- Knowledge of GPU virtualization and sharing technologies
- Experience building billing or metering systems
Details
- They offer competitive salary and equity based on experience and skillset
- Hybrid role: 3 days in office, 2 days WFH; must be located in Palo Alto
- Applicants must be authorized to work in the United States without visa sponsorship
How this job compares
Set against the 3,287 open Backend Engineer jobs on Olive Jobs, counted November 8, 2026.
- Pay
- This postingNot listed
- California median$170–247K/yr827 salary ranges
- US median$60–70/hr36 hourly ranges
- Openings
- Olive Jobs lists 1,082 open Backend Engineer jobs in California and 3,287 across the US. 14% of them nationwide are listed as remote or flexible.
- Experience
- This posting asks for at least 5 years of experience; Backend Engineer postings that state a minimum ask for 5 years at the median (2,701 postings).
- Education
- 809 of the 3,287 open Backend Engineer jobs (25%) ask for a bachelor's degree or higher.
Counted across the open, public US postings on Olive Jobs on November 8, 2026. Pay medians are the middle values of the low ends and of the high ends of postings that list both ends of a range in the same cadence (a yearly salary or an hourly rate); a range's midpoint is halfway between its ends. Pay and experience figures resting on fewer than 20 postings are left out.
Prepare for this role
You've read the whole posting — now see how you match it.
