Principal Systems Administrator

Staff HPC EngineerASRC Federal Data Solutions (AFDS) supports NASA's High Performance Computing Services program at the Ames Research Center in Mountain View, CA. Make a DIFFERENCE on a program that supports 4 on-site supercomputers, 18,000+ nodes, and 17+ combined petaflop supercomputer systems. Our program provides High Performance Computing services throughout the HPC lifecycle, including computational requirements, architecture, acquisition, and operations, for federal government customers.

Additionally, our team supports the Human-Systems Integration (HSI) Division, which enables the development of complex aerospace systems through analysis and modeling of human performance and human-automation interactions.Our employees embrace innovation and are committed to a culture of continuous, standards-driven process improvement and assimilation of industry's best practices. We have an immediate position for a Sr. Systems Administrator to join our Infrastructure team in Mountain View.

This role supports the hundreds of infrastructure systems that support HPC operations.Position Summary: The successful candidate will be an active supporting member of the ASRC Federal team reporting directly to the Manager of the HPC Computer Systems and Storage (CSS) group.Duties and Responsibilities:Maintains HPC clusters with over 8,000 nodes with InfiniBand, 100+ petabytes of data storage in production.InfiniBand fabric administration and operations Administer and maintain EDR/HDR InfiniBand fabrics, including switches, Host Channel Adapters (HCAs), cables, transceivers, and associated components.Monitor fabric health, link status, port errors, link speed, link width, congestion, and switch performance.Manage Subnet Manager (SM) services, including OpenSM or vendor-provided fabric management solutions.Maintain fabric topology, routing, addressing, partitioning, and configuration consistency.Identify and resolve failed links, degraded ports, disconnected nodes, and fabric initialization issuesPerformance monitoring and optimization Establish performance baselines for bandwidth, latency, throughput, and error rates.Diagnose packet drops, symbol errors, link recovery events, congestion, and performance degradation.Conduct bandwidth and latency testing using tools such as (I don't know)Optimize fabric routing, topology, and configuration to support MPI applications, parallel filesystems, and distributed HPC workloads.Linux, HCA, and software integration Install, configure, and maintain InfiniBand drivers, firmware, OFED or inbox RDMA software stacks, and associated libraries.Validate HCA compatibility with operating systems, kernels, switch firmware, and application software.Troubleshoot RDMA, verbs, MPI, and connectivity issues between compute nodes and storage platforms.Support integration with job schedulers, parallel filesystems, and HPC cluster provisioning tools.Automate health checks, configuration validation, and routine maintenance using Bash or Python and, where appropriate, Ansible.Assists in developing and writing proposals and publications.Creates and provides clear documentation.After hours/weekend support as requiredModerate Supercomputing System Administration that contributes to: Day-to-day operations of the Linux HPC clusters and storage systemsProactive monitoring, analyzing, and correct system issuesDevelopment of scripts to automate repetitive tasks or tools to enhance support of the HPC systemsSystem performance analysis and tuningBuilding, installing, and supporting user-requested softwareSupporting evaluation and assessment of new HPC technologyResolving user report issues and managing support tickets requests in RemedyRequired Qualifications:Bachelor's degree in computer science or related fieldStrong computer science background with in-depth systems-level knowledge in operating systems and networkingA minimum of 7 years' experience of administration of Linux systemsStrong ability to analyze and troubleshoot network issue with a strong understanding of HDR/EDR link speeds, HCAs, switch ports, LIDs, GUIDs, GIDs, subnet management, and fabric topologyStrong ability to analyze, debug and maintain the integrity of an existing code baseDemonstrated equivalence of 7+ years of Linux/UNIX user support experience and hands-on experience with administration of Linux systemsSuperior scripting skills and excellent attention to detail; proficiency in at least Python, Perl, or BashStrong ability to interact with customers to understand needs, elicit requirements, and get feedback on prototype solutionsExcellent communication and people skills; excellent time management and organizational skillsExperience with system configuration management tools e.g., ansible, puppet, chefExperience with revision control software e.g. CVS, SVN, GitProficiency at technical writingPreferred Qualifications:Proficiency with analysis and problem-solving skills for debugging and optimization of applicationsFamiliarity/proficiency with OpenMP and Message Passing Interface (MPI) programmingExperience with LustreExperience with cloud technologies (AWS, Azure, GCP), OpenStack or Kubernetes is a plus