Data Engineer / Enterprise Telemetry & Data Pipeline SME

San Antonio, Texas

Top Secret w/ SCI

On-site only

Information Technology

Overview

GovCIO is seeking a Senior Data Engineer / Enterprise Telemetry & Data Pipeline SME to design, optimize, automate, and sustain large-scale data collection and streaming pipelines supporting Air Force Intelligence Community missions.

The environment includes a global Elastic deployment with distributed endpoint collectors, syslog and telemetry from Cisco and other network devices, Cribl-based routing and normalization of native and non-native sources, Splunk engineering and integration, and AWS-based infrastructure and services. The primary focus is the movement and optimization of machine-generated data from source to authorized analytics and SIEM destinations.

This is not primarily a data science or dashboard-development role. The successful candidate must be able to evaluate the complete data path, identify collection and transport inefficiencies, engineer scalable solutions, automate repeatable operations, and independently resolve complex ingestion, routing, performance, reliability, and integration problems. This position will be located on Lackland AFB and will be an onsite position.

Responsibilities

• Serve as the senior technical owner and subject-matter expert for AF IC enterprise telemetry collection, ingestion, transformation, normalization, routing, and delivery architectures.

• Architect and optimize end-to-end data flows from distributed endpoints, servers, applications, network devices, security platforms, and cloud services into enterprise Elastic, Splunk, and other authorized analytics platforms.

• Assess collection architectures for bottlenecks, duplicate or unnecessary data, inefficient routing, bandwidth consumption, data-loss risk, processing overhead, and modernization opportunities.

• Engineer and troubleshoot Cribl Stream and Cribl Edge pipelines, including sources, destinations, routes, transformations, parsing, filtering, enrichment, normalization, buffering, and replay.

• Engineer Elastic collection and ingestion capabilities using Elastic Agent, Beats, Logstash, ingest pipelines, APIs, data streams, Fleet/integrations, and related technologies.

• Support Splunk ingestion and integration using forwarders, syslog, HTTP Event Collector, APIs, intermediate collection tiers, and other approved mechanisms.

• Develop data schemas, field mappings, metadata standards, tagging, enrichment, and normalization practices that improve interoperability and downstream usability.

• Design resilient pipeline patterns using buffering, queueing, retry, backpressure management, store-and-forward, failover, and recovery for geographically distributed and bandwidth-constrained environments.

• Establish pipeline observability and data-quality controls to identify dropped, delayed, malformed, duplicated, or incorrectly routed events and measure throughput, latency, queue depth, and error rates.

• Perform capacity and performance analysis across collectors, transport links, processing tiers, AWS services, and downstream analytics platforms; recommend changes that improve efficiency and scale.

• Optimize data volume and routing to reduce unnecessary network, compute, storage, cloud, and SIEM ingestion cost without compromising mission or cybersecurity requirements.

• Design and support AWS-based ingestion, streaming, processing, storage, and monitoring architectures appropriate to mission requirements.

• Develop Python tooling and automation for data transformation, pipeline validation, API integration, configuration generation, health monitoring, testing, and troubleshooting.

• Develop and maintain Infrastructure as Code and configuration automation using Terraform, CloudFormation, Ansible, or equivalent technologies.

• Manage pipeline code, configuration, and infrastructure artifacts in Git and integrate changes into CI/CD and DevSecOps workflows with automated validation and controlled promotion.

• Work with network, cloud, cybersecurity, systems, Elastic, Splunk, and DevSecOps teams to resolve cross-domain data-flow and infrastructure issues.

• Ensure data transport and pipeline implementations meet applicable security, access-control, encryption, certificate, auditing, retention, and compliance requirements.

• Lead root-cause analysis for data loss, latency, parsing, routing, ingestion, scaling, capacity, and integration failures and implement corrective actions that reduce recurrence.

• Develop architecture and data-flow diagrams, implementation plans, engineering standards, runbooks, SOPs, and technical recommendations.

• Provide technical leadership for telemetry modernization, platform integration, technology evaluation, migration, and enterprise scaling efforts.

Qualifications

High School with 10+ years (or commensurate experience)

Required Skills and Experience

Candidates must demonstrate substantial hands-on experience engineering production data pipelines. Data analysis, dashboard development, SIEM use, or product administration without ownership of collection and transport architecture is not sufficient for this position.

• Clearance: TS/SCI

• 10+ years of progressively responsible experience in data engineering, platform engineering, DevOps, systems engineering, software engineering, observability engineering, or closely related disciplines.

• 5+ years of hands-on experience designing, implementing, operating, or optimizing large-scale production data ingestion, logging, telemetry, event-streaming, or observability pipelines.

• Demonstrated experience serving as a senior engineer, technical lead, architecture resource, primary escalation point, or technical owner for production data platforms or pipelines.

• Strong understanding of distributed data-pipeline architecture, including collection, transport, buffering, queueing, parsing, transformation, normalization, enrichment, routing, storage, and downstream consumption.

• Demonstrated ability to diagnose and optimize throughput, latency, reliability, data quality, bandwidth utilization, processing efficiency, and pipeline resiliency.

• Significant hands-on production experience with Elastic ingestion and collection technologies such as Elastic Agent, Beats, Logstash, ingest pipelines, APIs, data streams, Fleet/integrations, or comparable mechanisms.

• Hands-on production experience with Cribl Stream, Cribl Edge, or equivalent enterprise routing/normalization technology, including building and troubleshooting sources, routes, pipelines, transformations, and destinations.

• Hands-on experience engineering or supporting Splunk ingestion and collection workflows using forwarders, syslog, HTTP Event Collector, APIs, or equivalent collection mechanisms.

• Experience ingesting and processing syslog, endpoint telemetry, network-device data, application logs, structured/unstructured events, API data, and other machine-generated sources.

• Strong knowledge of parsing, field extraction, schema design, normalization, enrichment, filtering, transformation, and routing.

• Demonstrated production experience with Amazon Web Services (AWS) and ability to design and troubleshoot AWS-based data-processing and telemetry architectures.

• Working knowledge of AWS services and patterns relevant to streaming, processing, storage, and monitoring, such as Kinesis, MSK/Kafka, S3, Lambda, EKS/ECS, CloudWatch, IAM, and VPC services.

• Strong Python programming and scripting experience used for automation, APIs, data manipulation, validation, testing, or engineering tools.

• Strong Linux systems engineering and troubleshooting skills and working knowledge of enterprise networking, including TCP/IP, DNS, routing, segmentation, TLS, proxies/firewalls, bandwidth, latency, and connectivity troubleshooting.

• Demonstrated use of Git for source control, peer review, branching, configuration management, and collaborative engineering workflows.

• Demonstrated experience with CI/CD or DevSecOps pipelines using GitLab CI/CD, Jenkins, GitHub Actions, AWS CodePipeline, Azure DevOps, or comparable technologies.

• Demonstrated Infrastructure as Code experience using Terraform, AWS CloudFormation, or equivalent technologies; experience with Ansible or comparable configuration automation.

• Working knowledge of cybersecurity requirements applicable to enterprise telemetry, including encryption, authentication, authorization, certificates, secrets, access control, data integrity, secure configuration, and change control.

• Demonstrated ability to independently investigate complex data-flow or integration problems, determine root cause, design remediation, implement and validate the solution, and document the outcome.

• Strong technical documentation and communication skills with the ability to coordinate effectively with Government stakeholders, cybersecurity, network, cloud, systems, DevSecOps, and analytics teams.

• Must meet IAT II 8570 certification requirement in lieu of defined 8140.2 defined work role.

Preferred Skills and Experience

The following capabilities distinguish especially strong candidates but are not substitutes for the required enterprise data-engineering depth above.

• Direct production experience with Elastic, Splunk, and Cribl operating together within the same enterprise telemetry architecture.

• Advanced Elastic ingestion experience, including large-scale Agent/Fleet, Logstash, data streams, index templates, lifecycle management, cluster ingestion behavior, and performance troubleshooting.

• Advanced Cribl engineering, including Stream/Edge, worker groups, routes, packs, replay, enterprise fleet management, performance tuning, and distributed processing patterns.

• Advanced Splunk ingestion experience, including Universal/Heavy Forwarders, HEC, syslog, deployment management, distributed collection, and indexer/data-flow architecture.

• Experience implementing Elastic Common Schema (ECS), Splunk Common Information Model (CIM), Open Cybersecurity Schema Framework (OCSF), or similar normalization standards.

• Experience with Kafka, Amazon MSK, Kinesis, or other enterprise event-streaming/message-broker technologies.

• Experience designing telemetry pipelines across globally distributed, classified, bandwidth-constrained, disconnected, degraded, or intermittently connected environments.

• Experience performing telemetry capacity planning using events/bytes per second, ingestion rate, retention, compression, network utilization, storage, and compute measurements.

• Experience optimizing SIEM or observability ingestion to reduce licensing, compute, storage, network, or cloud costs while preserving security and mission value.

• Experience with containerized or Kubernetes-based data processing, Amazon EKS, OpenTelemetry, Prometheus, Grafana, or comparable cloud-native observability technologies.

• Advanced AWS networking and security experience, including IAM, VPC design, private connectivity, load balancing, KMS/encryption, logging, and monitoring.

• Experience building automated schema validation, pipeline regression testing, end-to-end telemetry validation, GitOps, or automated configuration promotion.

• Experience integrating Cisco/network infrastructure, endpoint security products, vulnerability-management platforms, or other cyber telemetry into centralized analytics platforms.

• Experience supporting Zero Trust Architecture, Continuous Diagnostics and Mitigation (CDM), continuous monitoring, RMF, STIG, ATO, or enterprise cybersecurity analytics.

• Relevant technical certifications from AWS, Elastic, Splunk, Cribl, HashiCorp, Red Hat, Kubernetes, or cybersecurity certification programs.

#NSS #MAV #TMK #TLM

Posted Salary Range

USD $170,000.00 - USD $200,000.00 /Yr.

Company Overview

GovCIO is a team of transformers--people who are passionate about transforming government IT. Every day, we make a positive impact by delivering innovative IT services and solutions that improve how government agencies operate and serve our citizens.

But we can't do it alone. We need great people to help us do great things - for our customers, our culture, and our ability to attract other great people. We are changing the face of government IT and building a workforce that fuels this mission. Are you ready to be a transformer?

What You Can Expect

Interview & Hiring Process

If you are selected to move forward through the process, here’s what you can expect:

• During the Interview Process

• Virtual video interview conducted via video with the hiring manager and/or team

• Camera must be on

• A valid photo ID must be presented during each interview

• During the Hiring Process

• Enhanced Biometrics ID verification screening

• Background check, to include:

• Criminal history (past 7 years)

• Verification of your highest level of education

• Verification of your employment history (past 7 years), based on information provided in your application

Employee Perks

At GovCIO, we consistently hear that meaningful work and a collaborative team environment are two of the top reasons our employees enjoy working here. In addition, our employees have access to a range of perks and benefits to support their personal and professional well-being, beyond the standard company offered health benefits, including:

• Employee Assistance Program (EAP)

• Corporate Discounts

• Learning & Development platform, to include certification preparation content

• Training, Education and Certification Assistance*

• Referral Bonus Program

• Internal Mobility Program

• Pet Insurance

• Flexible Work Environment

*Available to full-time employees

Our employees’ unique talents and contributions are the driving force behind our success in supporting our customers, which ultimately fuels the success of our company. Join us and be a part of a culture that invests in its people and prioritizes continuous enhancement of the employee experience.

We are an Equal Opportunity Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, gender, gender identity or expression, sexual orientation, national origin, disability, or status as a protected veteran. EOE, including disability/vets.

Posted Pay Range

The posted pay range, if referenced, reflects the range expected for this position at the commencement of employment, however, base pay offered may vary depending on multiple individualized factors, including market location, job-related knowledge, skills, education, experience, and internal equity. The total compensation package for this position may also include other compensation elements, to be discussed during the hiring process. If hired, employee will be in an “at-will position” and the GovCIO reserves the right to modify base salary (as well as any other discretionary payment or compensation program) at any time, including for reasons related to individual performance, GovCIO or individual department/team performance, and market factors.