HPC Observability Engineer

22 hours ago Be among the first 25 applicants

Get AI-powered advice on this job and more exclusive features.

Direct message the job poster from EIT Professionals Corp

Role: HPC Observability Engineer (Python, HPC)

Location: Remote

Contract

Description:

The client has Grafana and InfluxDB services running on K8S in-house on-premises. Telegraf is used to ingest data from a GPU HPC cluster into InfluxDB. This engineer will help collect and visualize data for the “Terra” platform. The HPC Observability Engineer should have experience in:

• Setting up and maintaining Grafana dashboards for HPC environments
• Creating drill-down dashboards for servers, including metrics like memory, network, and CPU utilization
• Exploring and utilizing out-of-the-box metrics from InfluxDB
• Writing Python scripts for data ingestion into InfluxDB with examples
• Developing a proof of concept with a simple Python script to monitor load
• Ingesting Infiniband packet data
• Monitoring LSF jobs in various states
• Visualizing server-specific and cluster-wide metrics in Grafana
• Optional: Integrating third-party plugins like DDN’s Lustre, Mellanox fabric, etc.

Qualifications and Skills:

• B.Tech, MS, or PhD in Computer Science or related field
• 5-8 years of experience with Grafana, InfluxDB, and Telegraf
• Experience in Python and Bash scripting is a plus
• Knowledge of Docker and Google Cloud Platform is advantageous
• HPC operations experience is beneficial
• Strong communication skills and ability to work independently
• Proficiency in requirements analysis and automated testing
• Ability to write efficient, secure, and well-documented Python code
• Experience with Git and pipeline development
• Awareness of modern security and development practices

Responsibilities:

• Develop and leverage Grafana dashboards and Telegraf configurations
• Create dashboards for server and cluster metrics
• Develop Python scripts for data ingestion and documentation
• Visualize non-native resources in Grafana
• Optional: Integrate third-party plugins
• Maintain high-quality code and documentation
• Collaborate with teams to troubleshoot and optimize pipelines

Desired Skills:

• Python (good to have)
• Bash scripting (good to have)
• Docker (must)
• HPC operations and LSF (good to have)
• Experience with DDN Lustre, Mellanox fabric (good to have)
• Google Cloud Platform (good to have)
• Knowledge of Git (must)

Seniority level:

• Mid-Senior level

Employment type:

• Contract

Job function:

• Engineering and Information Technology

Industries:

• IT Services and IT Consulting

This job is active and accepting applications.