STAFF SOFTWARE ENGINEER

Key Responsibilities:

• Develop, deploy, operate, and maintain infrastructure platform software for Site Reliability Engineering, with a focus on ensuring the observability, availability, and operational efficiency of enterprise retail store systems.

• Build and extend internal reliability platforms deployed on Google Cloud Platform and Google Kubernetes Engine, using infrastructure-as-code tools including Terraform and cdk8s (Cloud Development Kit for Kubernetes written in TypeScript) to provision and manage cloud resources and Kubernetes workloads programmatically.

• Deploy, configure, tune, and maintain open-source observability systems including Grafana, Loki (log aggregation), Tempo (distributed tracing), Prometheus (metrics collection), Pyroscope (continuous profiling), and AlertManager (alert routing) on Google Kubernetes Engine clusters.

• Write queries in PromQL, LogQL, and TraceQL to build dashboards, configure alert rules, and troubleshoot production infrastructure issues.

• Define and implement Service Level Objectives (SLOs), Service Level Indicators (SLIs), and error budgets using Prometheus-based SLI queries and multi-window, multi-burn-rate alerting configurations for monitoring across multiple application services.

• Develop and maintain operational automation platform infrastructure, including provisioning and managing automation runners on both Kubernetes (via cdk8s) and virtual machines (via Terraform), developing integration plugins, managing access control lists through GitHub-backed configuration, and resolving platform support requests.

• Build automation to reduce operational toil, including self-service onboarding tools, GitHub Actions CI/CD pipelines for infrastructure deployments, and AI-assisted workflows using GitHub Copilot.

• Participate in on-call rotation for observability and automation infrastructure.

• Investigate and resolve production incidents affecting logging, tracing, profiling, and automation platforms.

• Contribute to blameless post-mortem reviews and implement systemic improvements.

• Validate platform resilience through destructive and performance testing.

• Document platform operations, create runbooks, and maintain onboarding guides for internal engineering teams.

• Telecommuting available.

Minimum Qualifications:

• Must be eighteen years of age or older.

• Must be legally permitted to work in the United States.

• Bachelor's degree in Computer Science, Computer Engineering, Information Technology Management, Electronics Engineering, or a related field and 3 years of experience in a related occupation. Requires 3 years of experience in the following skills:

• Deploying, configuring, and troubleshooting containerized workloads on Google Kubernetes Engine (GKE) using infrastructure-as-code tools including cdk8s for programmatic TypeScript-based generation of Kubernetes manifests and Helm charts for templated Kubernetes deployments.

• Provisioning and managing cloud infrastructure on Google Cloud Platform (GCP) using Terraform, including Google Kubernetes Engine clusters, Cloud Storage buckets, BigQuery datasets, Cloud Pub/Sub topics and subscriptions, Cloud SQL instances, Cloud Logging sinks, Compute Engine instances, Identity and Access Management (IAM) policies, and Workload Identity federation.

• Deploying, operating, and tuning open-source observability platforms on Kubernetes, specifically Loki (log aggregation with LogQL), Tempo (distributed tracing with TraceQL), Prometheus (metrics collection with PromQL), Pyroscope (continuous profiling), and AlertManager (alert routing and silencing).

• Defining and implementing Service Level Objectives (SLOs), Service Level Indicators (SLIs), and error budgets using Prometheus-based SLI queries, multi-window multi-burn-rate alerting for monitoring frameworks configured through YAML definitions.

• Building and maintaining operational automation platform infrastructure, including provisioning automation runners on Kubernetes and virtual machines, developing platform integration plugins, managing GitHub-backed access control and runbook configurations, and resolving platform support requests.

• Programming in Python, TypeScript, or similar for infrastructure automation, platform tooling, observability configuration, and Kubernetes manifest generation.

• Designing and operating synthetic monitoring platforms deployed to Kubernetes clusters, including test orchestration, result collection via Cloud Pub/Sub, and alerting integration.

• Configuring Kubernetes security controls including Role-Based Access Control (RBAC), namespace isolation, service accounts, and GCP IAM policies including Workload Identity for secure service-to-service authentication.

• Building CI/CD pipelines using GitHub Actions for automated testing, building, and deployment of infrastructure platform services, including container image builds, Helm chart packaging, and Terraform plan/apply workflows.

• Building automation to reduce operational toil in infrastructure operations, including self-service onboarding tools, AI-assisted workflows using GitHub Copilot, and automated support request handling.

• Troubleshooting production infrastructure issues across distributed systems running on Kubernetes, including diagnosing pod failures, resource contention, network policies, storage issues, and observability pipeline bottlenecks using kubectl, log analysis, and tracing data.

• Managing secret and credential lifecycle using Google Secret Manager, Kubernetes secrets, and automated credential rotation for platform integrations. We are an Equal Opportunity Employer and do not discriminate against any employee or applicant for employment because of race, color, sex, age, national origin, religion, sexual orientation, gender identity, status as a veteran, and basis of disability or any other federal, state or local protected class.