Sr Systems Development Engineer, AWS AI/ML Servers

Systems Development EngineerAWS runs the world's largest fleet of AI/ML accelerator servers. When a model with billions of parameters trains across a large scale of GPUs, every minute of downtime costs real progress. We are building the automation, diagnostics, and predictive intelligence that keeps this fleet running at peak. If you want to work at the intersection of hardware, software, and scale — where your code directly prevents customer-impacting failures — this is the role.We are seeking a Systems Development Engineer to build automation software, diagnostic tooling, and fleet health infrastructure for our accelerated compute platforms. You will work across multiple teams and organizations to design scalable, reliable systems for our accelerated compute fleet.What You Will DoTackle problems no one has fully defined yet — spanning hardware, firmware, kernel, and software simultaneously.Own systems end to end, writing code that prevents failures rather than reacts to them, and building automation that replaces manual toil with intelligent self-healing.Work across PCIe topology, GPU diagnostics, Linux drivers, and telemetry pipelines to correlate signals and isolate faults at fleet scale.Why You Will Love ItYour automation runs at a large scale across servers in the cloud. When you ship, you see failure rates move within days. The team is small enough that your decisions shape the architecture, and large enough that you will always have experts to learn from across hardware, firmware, and software.The Ideal CandidateYou know the full stack from bare-metal to userland. You debug at the intersection of components, not just within them. You build at cloud scale and care how your systems decisions impact customers. You are an excellent communicator who can drive alignment across hardware, software, and operations teams.Key job responsibilitiesFleet Health & Predictive InfrastructureBuild and own the automation infrastructure for accelerator (AI/ML) fleet health at a large scale of servers, driving toward zero-touch operations that detect, diagnose, triage, and remediate faults without human interventionDesign and develop test frameworks, test coverage strategies, and diagnostic tooling to validate hardware functionality, detect faults, and ensure qualification coverage across the platform lifecycle.Design predictive failure detection using telemetry, sensor data, error trending, and log correlation to identify degrading components before customer impactDevelop monitoring dashboards and alerting for real-time fleet health visibility across manufacturing, lab, and production environmentsDefine and track fleet health metrics: failure rates, mean time to detect and resolve issues, first-time fix rate, test dwell time, and predictive accuracyDebugging & TroubleshootingDebug complex system-level issues across compute, GPU, and networking in production — including Linux boot/runtime failures, PCIe, power, NIC, NVMe, and GPU subsystems on x86 and ARMPerform root cause analysis correlating across firmware, kernel, driver, and physical layer; feed findings into manufacturing quality and design improvementsSystems Development & AutomationDesign scalable test automation for hardware bring-up, regression, and qualification — reducing manufacturing test cycle times without sacrificing coverage through intelligent test sequencing and parallel executionBuild data pipelines correlating test results, sensor telemetry, and component-level data to identify systemic yield issues and drive upstream fixesDevelop and maintain Linux device drivers on ARM and x86; work with OS internals and accelerator/GPU software stacksBuild and manage tests covering all functional aspects of the system and CI/CD pipelines for rapid deployment to manufacturing lines and production fleetCross-Team CollaborationWork across engineering teams and internal customers to ensure new accelerated compute hardware meets data path, control path, and onboarding requirementsEngage with ODMs and design partners on testability, diagnostic coverage, and automation requirements during hardware design and bring-up phases — influencing functional and performance readiness of the platformPartner with datacenter operations to close the loop between field failures, manufacturing escapes, and design improvementsOperational ExcellenceParticipate in post-incident reviews, identify contributing causes and drive permanent fixes that eliminate whole classes of riskProduce clear, maintainable documentation for systems, runbooks, and automation to enable others to operate and extend your workDrive process improvements that increase team agility — reducing development friction, eliminating unnecessary gates, and improving delivery velocityMay require occasional (<10%) regional and international travel to Design and Manufacturing Partner sites.A day in the lifeYou start the day reviewing overnight validation run results, triaging a cluster of GPU errors that correlate with a specific firmware version. Mid-morning, you push a fix to your diagnostic automation pipeline and validate it catches the failure pattern in your test environment. In the afternoon, you join a hardware bring-up call with your ODM partner to debug a PCIe link training failure on a new EVT board, walking the team through kernel logs and signal integrity data. You end the day reviewing a pull request from a teammate on a new telemetry correlation engine, and updating your manufacturing test coverage dashboard with the latest yield data.About the teamThe Hardware Engineering AI/ML UltraServer platform team is a group of engineers and technical program managers directly responsible for launching GPU-accelerated servers into the AWS fleet. Located in Seattle, Austin, and Cupertino, we collaborate with global development teams and ODM partners to deliver next-generation AI/ML infrastructure deployed in datacenters worldwide. We move fast with small, empowered teams delivering end-to-end — from server conception through fleet-scale operations.Basic Qualifications6+ years of systems design, software development, operations, automation, and process improvement experience6+ years of non-internship professional software development experience6+ years of programming with at least one modern language such as C++, C#, Java, Python, Golang, PowerShell, Ruby experience6+ years of designing or architecting (design patterns, reliability and scaling) of new and existing systems experienceBachelor's degree in Computer Science, Computer Engineering, Electrical Engineering, or equivalent work experience5+ years of Linux/Unix operating systems experiencePreferred QualificationsMaster's degree or above in computer science, electrical engineering, or related field5+ years of full software development life cycle, including coding standards, code reviews, source control management, build processes, testing, and operations experienceExperience taking a leading role in building complex software or computing infrastructure that has been successfully delivered to customers5+ years of building scripts, tooling, and automation for large-scale computing environments experienceExperience with general troubleshooting/debugging of hardware, or experience with CUDA kernels or ML/low-level kernels5+ years of hardware design and validation of components, subsystems and systems experienceExperience leading technical teams through daily operations and maintenance evolutions, or experience in data center engineering or operations