DevOps EngineerWe're building infrastructure to test and secure AI systems under extreme, adversarial, and failure-prone conditions. From custom CUDA kernels and trusted execution environments (TEEs) to zero-knowledge proof systems, we design for correctness and resilience even when things go catastrophically wrong.We want to simulate what happens when systems face the worst: thermal stress on GPUs, cosmic-ray-like memory faults, and root-level adversaries targeting cryptographic protocols. If a failure mode could take down an AI system in production, we aim to surface it first under controlled, repeatable conditions.We're looking for a DevOps Engineer to help us build and operate high-integrity, security-conscious testing infrastructure that directly informs production hardening.
You'll work across hardware and software layers to design systems that remain observable, testable, and verifiable while under attack or extreme stress; taking findings from our test set ups directly into our production deployments that secure statistical models and AI systems for our users.What You'll DoArchitect and maintain automated adversarial testing environments where attackers have full host or root access, then translate findings into updates to our products.Build and manage hardware-in-the-loop test setups, including environmental chambers and stress rigs for GPUs, to validate the production resilience of our fault tolerant CUDA kernels.Develop fault injection frameworks simulating everything from bit flips and power loss to protocol-level faults that could occur in production.Implement CI/CD pipelines for our core products, ensuring their integrity with every new code change.Manage the infrastructure that manages TEE attestations, zero-knowledge proof creation and delivery, and fault-tolerant AI inference for our users.Design observability, monitoring, and alerting systems that work in both intentionally unstable test environments and hardened production systemsWhat We're Looking ForMindset & ApproachA security-first mentality: you think like an attacker to build better defenses,Strong bias toward reproducibility, security, and traceability in complex environments.Comfortable working in ambiguous and high-failure environments where resilience matters.Core SkillsProficiency with CI/CD systems (GitLab CI, Jenkins, Buildkite) and Infrastructure as Code tools (Terraform, Ansible, Pulumi).Experience with container orchestration (Docker, Kubernetes) and building reproducible environmentsStrong Linux systems knowledge, especially around debugging, performance, and kernel behaviorExpertise with observability and monitoring tools like Prometheus, Grafana, ELK, or OpenTelemetry in both test and production contextsSystems & HardwareExperience with GPU-based compute (NVIDIA stack, CUDA, thermals, memory behavior)Comfort working with bare-metal or lab hardware (rack-mounted systems, thermal chambers, environmental sensors)Background in systems-level engineering or reliability, especially at hardware/software fault boundariesSecurity & Adversarial ThinkingExperience with secure systems design and threat modeling at the infrastructure levelAbility to simulate adversarial scenariosFault InjectionKnowledge of chaos engineering practices, fault injection frameworks (Chaos Mesh, Gremlin), or fuzzing toolsAbility to design hostile test conditions that replicate real-world production failures (power fluctuations, silent data corruption)Bonus SkillsFamiliarity with AI/ML infrastructure (model serving, distributed training, inference under load)Comfort working with ECC error simulation, or low-level hardware errorsExperience securing production AI/ML systems against adversarial attacksFamiliarity with zero-knowledge proof systems, cryptographic verification, or TEEs (SGX, SEV, TrustZone).This role is perfect for someone who loves breaking things to make them unbreakable, and who understands that the best production security comes from testing systems to their absolute limits.