Skip to main content
Clockwork Systems logo

Senior Software Engineer – Network Observability

Clockwork Systems

Palo Alto, CAJob$140–210K/yrPosted 5 days agoStill listed 4 days ago

See where you stand before you apply. No sign-up to start.

You apply on the posting's application page, with no Olive account needed.

Watch jobs like this.

Don't just apply. Show up ready.

Olive works from this exact posting.

At a glance

Compensation
$140–210K/yr
Location
Palo Alto, CA
Work Authorization
Not specified

Olive focuses on jobs in the United States, including remote roles you can work from there.

Job overview

Clockwork Systems is seeking a Senior Software Engineer to architect, develop, and scale a high-performance network monitoring and observability platform. The role focuses on visibility into RDMA, RoCE, InfiniBand, and TCP/IP networks, including telemetry services, diagnostics, anomaly detection, and traffic analysis. The engineer will troubleshoot production issues across software and infrastructure layers and collaborate on scalable solutions and the platform’s technical direction.

Skills & qualifications

RequiredNice to have

Skills

Google Cloud Platform, nice to haveData Visualization, nice to haveRust, nice to haveMachine Learning, nice to haveLinux, nice to haveStorage, nice to haveDatadog, nice to haveRemote Direct Memory Access, nice to haveArtificial Intelligence, nice to haveSecurity Compliance, nice to haveSplunk, nice to haveLinux Kernel, nice to haveInfrastructure Automation, nice to haveFlow Control, nice to haveOperating Systems, nice to haveDiagnostics, nice to haveCloud Infrastructure, nice to haveTCP/IP, nice to haveDistributed Systems, nice to haveGrafana, nice to haveWireshark, nice to havePerformance Tuning, nice to haveInfrastructure, nice to haveObservability, nice to haveGPU, nice to haveKubernetes, nice to havePacket Loss, nice to haveMicrosoft Azure, nice to haveTcpdump, nice to haveDNS, nice to haveGolang, nice to haveService, nice to haveConfiguration Management, nice to haveAmazon Web Services, nice to haveNetworking Technologies, nice to haveHigh Performance Computing, nice to havePrometheus, nice to haveCluster Management, nice to havePython, nice to haveSystems Programming, requiredInfrastructure Projects, requiredBackend Services, requiredTelemetry Pipelines, requiredObservability Platforms, requiredHigh-Performance Networking, requiredRDMA Concepts, requiredLinux Networking, requiredHTTP, requiredRouting, requiredMTU, requiredCongestion Control, requiredLatency, requiredNetwork Diagnostics, requiredPath Discovery, requiredNetwork Reachability, requiredSynthetic Probes, requiredActive Network Measurements, requiredMonitoring Platforms, requiredVisualization Platforms, requiredDebugging, requiredProduction Systems, requiredTechnical Judgment, requiredReliable Systems, requiredScalable Systems, requiredMaintainable Systems, requiredAI/ML Infrastructure, nice to haveHPC Infrastructure, nice to haveStorage Infrastructure, nice to haveGPU Cluster Infrastructure, nice to haveRoCE Deployments, nice to haveInfiniBand Deployments, nice to haveNCCL, nice to haveDistributed Training, nice to have

Qualifications

Bachelor's or Master's in Related Field, required

Full job description

About Clockwork Systems

Clockwork.io – Software Driven Fabrics to increase GPU cluster utilization

Clockwork Systems was founded by Stanford researchers and veteran systems engineers who share a vision for redefining the foundations of distributed computing. As AI workloads grow increasingly complex, traditional infrastructure struggles to meet the demands of performance, reliability, and precise coordination. Clockwork is pioneering a software-driven approach to AI fabrics by delivering cross-stack observability to catch and quickly resolve problems, workload fault tolerance to keep jobs running through failures, and performance acceleration that dynamically routes and paces traffic to avoid congestion.

To learn more, visit www.clockwork.io.

About the Role

We are seeking an experienced Senior Software Engineer to contribute to the architecture, development, and scaling of a high-performance network monitoring and observability platform. This role will focus on building systems that provide deep visibility into RDMA, RoCE, InfiniBand, and TCP/IP networks. The ideal candidate has strong experience in distributed systems, Linux networking, and modern observability stacks (e.g., Grafana/Prometheus).

What You Will Do

  • Design, develop, and scale high-performance network monitoring platforms for RDMA, RoCE, InfiniBand, and TCP/IP infrastructure.

  • Build backend telemetry services, observability dashboards, alerts, diagnostics, anomaly detection, SLA monitoring, and traffic analysis workflows.

  • Troubleshoot complex production issues across application, OS, server, RDMA, and network layers while optimizing low-latency collection, aggregation, and alerting.

  • Collaborate with cross-functional teams to deliver scalable solutions, improve engineering practices, automate operational workflows, and contribute to the technical direction of the platform.

What We’re Looking For

  • Bachelor's or Master's degree in Computer Science, Computer Engineering, Electrical Engineering, or a related technical field.

  • Strong hands-on programming experience in C++, Go, Python, Rust, or similar systems programming languages.

  • Proven experience delivering complex infrastructure projects and contributing to the design and implementation of distributed systems.

  • Experience building distributed systems, backend services, telemetry pipelines, or observability platforms.

  • Hands-on experience with RDMA, RoCE, InfiniBand, or other high-performance network fabrics.

  • Familiarity with libibverbs, RDMA verbs, RDMA CM, queue pairs, completion queues, memory registration, and related RDMA concepts.

  • Strong knowledge of Linux networking, TCP/IP, DNS, HTTP, routing, MTU, congestion control, packet loss, latency, and performance tuning.

  • Experience with traceroute-style diagnostics, path discovery, network reachability checks, synthetic probes, or active network measurements.

  • Experience with monitoring and visualization platforms such as Prometheus, Grafana, Datadog, Splunk, OpenTelemetry, or similar tools.

  • Strong debugging skills across software, operating system, server, and network layers.

  • Experience operating production systems in Linux-based environments.

  • Strong technical judgment and ability to build reliable, scalable, and maintainable systems.

Nice to Have

  • Experience supporting AI/ML, HPC, storage, or GPU cluster infrastructure workloads.

  • Experience with large-scale RoCE or InfiniBand deployments.

  • Experience with NCCL, distributed training infrastructure, or AI cluster diagnostics.

  • Experience with eBPF, XDP, DPDK, perf, tcpdump, Wireshark, ethtool, iproute2, rdma-core, or Linux kernel networking tools.

  • Experience with cloud infrastructure on AWS, GCP, or Azure.

  • Experience with Kubernetes, service discovery, configuration management, and infrastructure automation.

  • Knowledge of security, compliance, and infrastructure best practices.

  • Experience designing time-series data systems, alerting pipelines, or high-cardinality telemetry platforms.

Enjoy

  • Challenging projects.

  • A friendly and inclusive workplace culture.

  • Competitive compensation.

  • A great benefits package.

  • Catered lunch.

Compensation for this position will vary based on the skills and experience you bring, as well as internal equity considerations. For candidates hired at the posted level, the expected base salary range is $140,000 - $210,000. The offered compensation package may also include stock options or other equity awards, subject to Clockwork’s equity program and applicable approvals. In addition to cash compensation, this role is eligible to participate in the company’s equity program, which may include stock options granted in accordance with the company’s equity plan and subject to approval and applicable vesting schedules. Clockwork Systems is an equal opportunity employer. We are committed to building world-class teams by welcoming bright, passionate individuals from all backgrounds. All qualified applicants will receive consideration for employment without regard to race, color, ancestry, religion, age, sex, sexual orientation, gender identity or expression, national origin, disability, or protected veteran status. We believe diversity drives innovation, and we grow stronger together.

How this job compares

Set against the 9,067 open Software Engineering jobs on Olive Jobs, counted November 15, 2026.

Pay
  • This posting$140–210K/yr
  • California median$189–280K/yr1,413 salary ranges
  • US median$143–250K/yr7,144 salary ranges

This range's midpoint, $175K, is 24% below the California median midpoint ($232K) — in the lowest quarter of Software Engineering salary ranges in California.

Openings
Olive Jobs lists 1,892 open Software Engineering jobs in California and 9,067 across the US. 9% of them nationwide are listed as remote or flexible.
Hiring the most in California
Experience
Software Engineering postings that state a minimum ask for 7 years of experience at the median; the middle half ask for 5 to 10 years (6,824 postings).
Education
2,061 of the 9,067 open Software Engineering jobs (23%) ask for a bachelor's degree or higher.

Counted across the open, public US postings on Olive Jobs on November 15, 2026. Figures based on too few postings are left out.

Prepare for this role

You've read the whole posting — now see how you match it.