
Member of Technical Staff (Index, Search Crawler)
Dublin, CAFull-timePosted 3 days agoStill listed · checked in the last 2 days
See where you stand before you apply. No sign-up to start.
You apply on the posting's application page, with no Olive account needed.
Watch jobs like this. New roles like this one near Dublin, CA, by email.
Don't just apply. Show up ready.
Olive works from this exact posting.
At a glance
Olive focuses on jobs in the United States, including remote roles you can work from there.
Job overview
Perplexity is seeking an engineer to build and operate the indexing and data-processing layer of its Search Crawler. The role focuses primarily on offline and nearline pipelines that ingest, process, and organize crawled content into efficient, queryable structures. The engineer will work across indexing, storage, reliability, observability, and controlled experimentation, partnering with retrieval, ranking, and machine-learning teams to keep data fresh, complete, and accessible.
Skills & qualifications
Skills
Qualifications
Full job description
AI answers are only as trustworthy as the information behind them. Perplexity grounds its answers in fresh, real-world sources, and the index is what makes that possible. We're looking for an engineer to work on the indexing and data processing layer of Perplexity's Search Crawler. You'll build systems that ingest, process, and organize large volumes of crawled content into efficient, queryable structures, working primarily on offline and nearline pipelines that keep data fresh, complete, and accessible to downstream retrieval. Throughput, scalability, and correctness are critical across datasets that change continuously.
Responsibilities
-
Design, implement, and operate large-scale indexing systems and data pipelines at the core of Perplexity's search infrastructure.
-
Develop and optimize indexing strategies that balance performance, freshness, and resource efficiency.
-
Work on storage formats, compaction, and update mechanisms that keep data accessible and current.
-
Keep pipelines reliable and predictable under high-throughput conditions.
-
Build well-tested components with clear responsibilities and interface contracts that stay flexible as the system evolves.
-
Define and implement observability across offline and near line pipelines, including structured logs, metrics, and data quality signals.
-
Monitor throughput, resource usage, and cost, and drive optimisations as business needs change.
-
Partner with retrieval, ranking, and ML teams to make sure indexing outputs meet their requirements.
-
Enable safe experimentation on indexing strategies and processing logic through controlled rollouts and clearly defined quality signals.
Qualifications
-
5+ years of experience building production backend or data infrastructure systems.
-
Strong programming experience in Go, C++, or Rust.
-
Experience with high-throughput data processing systems and very large datasets.
-
Experience building or operating databases, storage systems, data planes, or indexing pipelines.
-
Strong understanding of distributed systems, fault tolerance, consistency, and scalability.
-
Experience running production systems and handling operational incidents.
-
A systems-thinking mindset and the ability to reason about end-to-end data flows.
How this job compares
Set against the 36,943 open Software Engineer jobs on Olive Jobs, counted November 20, 2026.
- Pay
- This postingNot listed
- California median$161–243K/yr6,680 salary ranges
- California median$48–67/hr122 hourly ranges
- Openings
- Olive Jobs lists 8,260 open Software Engineer jobs in California and 36,943 across the US. 8% of them nationwide are listed as remote or flexible.
- Experience
- This posting asks for at least 5 years of experience; Software Engineer postings that state a minimum ask for 5 years at the median (30,715 postings).
- Education
- 10,965 of the 36,943 open Software Engineer jobs (30%) ask for a bachelor's degree or higher.
Pay, requirements and demand for Software Engineer jobs
Counted across the open, public US postings on Olive Jobs on November 20, 2026. Figures based on too few postings are left out.
Prepare for this role
You've read the whole posting — now see how you match it.


