Principal Software Engineer
Location not statedJobPosted 6 days agoStill listed 6 days ago
See where you stand before you apply. No sign-up to start.
You apply on the posting's application page, with no Olive account needed.
Watch jobs like this. New roles like this one, by email.
Don't just apply. Show up ready.
Olive works from this exact posting.
At a glance
Olive focuses on jobs in the United States, including remote roles you can work from there.
Job overview
The Principal Software Engineer designs, builds, and operates highly available distributed services on the Substrate platform. The role focuses on storage and data-access services, ingestion and propagation pipelines, search freshness, scale and capacity, production reliability, and observability. It also raises engineering quality, applies automation and AI-assisted tooling to operations, and partners with service teams across regions on milestones, workloads, and architectural changes.
Skills & qualifications
Skills
Qualifications
Full job description
Design, build and operate highly available distributed services on the Substrate platform, with clear ownership for reliability, performance, security and cost. Evolve storage and data-access services, including client SDKs and APIs, migration from legacy access paths to supported clients, and correctness guarantees for reads, writes, deletes and queries. Build and modernize large-scale ingestion and propagation pipelines, including event-driven processing, queueing, prioritization, retry, flow control and backpressure, so that content and permission changes reach downstream systems quickly and reliably. Improve search freshness and completeness by tracing data end to end, detecting gaps, and building safe, replay-able and idempotent repair and backfill mechanisms. Drive scale and capacity work across shards and stamps (groups of shards managed as a unit), including scale-out, scale-up, rebalancing, in-place repartitioning and approaches that avoid costly full re-crawls of source data. Diagnose and mitigate production issues involving throttling, CPU, memory and disk pressure, hot spots, regional capacity constraints and cross-service dependencies, then deliver sustainable fixes. Build observability: metrics, logs, traces, dashboards, alerts and data-quality pipelines that distinguish real customer impact from reporting gaps. Raise engineering quality through design reviews, automated and functional testing, safe flighting and rollback, and reduction of test flakiness and toil. Apply automation and AI-assisted tooling to incident triage, diagnostics and routine operations. Partner with service teams across regions and time zones to plan milestones, onboard new workloads and land cross-team architectural changes. Bachelor's Degree in Computer Science or related technical field AND 6+ years technical engineering experience with coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, or Python OR equivalent experience. These requirements include but are not limited to the following specialized security screenings: Bachelor's degree in Computer Science or a related technical field, or equivalent practical experience. Typically 5+ years of professional software engineering experience for Senior, and 8+ years for Principal, building and shipping backend or infrastructure software. Equivalent demonstrated impact is considered. Solid coding skills in at least one systems or backend language such as C#, C++, Java, Go or Rust. Experience with C# and .NET is especially relevant; equivalent experience transfers well. Solid foundations in data structures, algorithms, concurrency, networking and operating systems. Experience designing and operating distributed systems or cloud services, including reasoning about failure modes, consistency, latency and throughput. Experience designing APIs or SDKs, writing testable code and building automated tests at unit, integration and end-to-end levels. Experience running production services, including participation in on-call rotations, incident response and root-cause analysis. Clear written and verbal communication in English, including design documents and status updates for technical and leadership audiences. 8+ years building large-scale distributed storage, database, search or messaging systems (Principal), or 5+ years with significant ownership in one of these areas (Senior). Hands-on experience with storage engines, distributed file or object stores, key-value stores, or search engines such as Lucene-based systems. Experience operating services at the scale of millions of tenants or billions of items, including multi-region and sovereign-cloud deployments. Experience leading a major migration, re-architecture or modernization of a production system with no customer-visible disruption. Track record of publications, patents, open-source contributions or internal technical talks in distributed systems or storage.
Prepare for this role
You've read the whole posting — now see how you match it.



