Network Sustainment Engineer — Night Software Upgrades

Job Description

Overview We are seeking a detail-oriented Network Sustainment Engineer to join our Sustainment team on a night-shift schedule, executing software upgrades across a large-scale, multi-platform network fleet. This role operates within tight maintenance windows to upgrade production network devices — including Cisco IOS-XE, IOS-XR, and Nexus platforms — following approved Method of Procedures (MOPs). The engineer works closely with lead engineers and hands off a clear outcome report to the day team at shift end. A strong ability to execute precisely under time pressure and escalate quickly when issues arise is essential.

Key Responsibilities

• Execute software upgrades on production Cisco network devices (IOS-XE, IOS-XR, and Nexus platforms) during overnight maintenance windows
• Follow multi-hop upgrade paths, navigating intermediate code versions as required by platform upgrade sequences (e.g., two- or three-step code progressions)
• Perform pre-upgrade health checks: validate console access, confirm device reachability, verify traffic state, and confirm MOP readiness before any change begins
• Execute MOPs with precision — command by command — and perform in-MOP validation checks at each gate
• Monitor device behavior during upgrade: reload progress, post-boot configuration integrity, routing adjacency restoration, and traffic resumption
• Perform post-upgrade validation using CLI outputs and internal tooling to confirm the device is fully healthy before closing the maintenance window
• Identify and triage issues encountered mid-window: determine whether to back out, hold, or escalate to a lead engineer
• Execute graceful back-out procedures when upgrades cannot be completed within the maintenance window
• Document all outcomes — successes, failures, back-outs — clearly in the shift handoff for the day remediation team
• Coordinate with the Network Operations Center (NOC) and on-call engineering support as needed during the window
• Flag console access issues, device unreachability, or configuration anomalies encountered prior to upgrade attempts

Day-to-Day Work

• Review assigned device list and MOPs before each shift; confirm scheduled maintenance windows and any market-specific constraints
• Validate console access and device health for each device in the night's queue before the maintenance window opens
• Execute upgrade MOPs sequentially across assigned devices, completing pre-, during-, and post-change validation at each step
• Monitor traffic failover behavior and routing protocol reconvergence following device reloads
• Use CLI outputs and internal tooling to confirm software version, configuration state, and network health post-upgrade
• Capture and document outcomes for each device: upgraded successfully, back-out executed, or failed with details
• Produce a clear end-of-shift handoff report detailing completed upgrades, outstanding failures, and any issues requiring day-team follow-up
• Communicate in real time with lead engineers for escalations, and with the NOC for any service impact during the window

Skills and Requirements

• Hands-on experience executing network changes in production environments, including during live maintenance windows
• Strong CLI experience on Cisco routers and switches (IOS-XE, IOS-XR, or Nexus)
• Experience performing and validating software upgrades or configuration changes on network devices
• Ability to read, interpret, and precisely execute Method of Procedures (MOPs)
• Proven ability to troubleshoot issues in real time under tight maintenance window constraints
• Strong understanding of network fundamentals: routing protocols, traffic flow, failover behavior, and device reload sequences
• Comfortable working an overnight shift schedule with consistent reliability
• Experience with multi-hop software upgrade paths and intermediate code validation steps
• Familiarity with console access methods and troubleshooting console connectivity issues (CSM/out-of-band access)
• Experience with Cisco Nexus platforms and NX-OS in addition to IOS-XE or IOS-XR
• Experience in telecom or other high-availability, mission-critical network environments
• Exposure to internal network validation or automation tooling
• Experience working within change control and freeze-window constraints

We are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal employment opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment without regard to race, color, ethnicity, religion, sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military or uniformed service member status, or any other status or characteristic protected by applicable laws, regulations, and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or the recruiting process, please send a request to [email protected].