Principal AI Engineer (Technical Lead)

PRINCIPAL AI ENGINEER Technical Lead San Francisco, CA • Hybrid • $250K-$350K

Join the Data Revolution at Prophecy

About Prophecy Prophecy is revolutionizing enterprise data transformation with AI-native tools that turn data chaos into reliable insights. We're not just building software—we're reshaping how the world's leading companies work with data. Our AI agents collaborate seamlessly with humans through intuitive visual and document interfaces, empowering teams to analyze, transform, and prepare data at enterprise scale.

The Role As Principal AI Engineer (Technical Lead), you'll be at the forefront of AI innovation. This is a hands-on leadership role where you'll architect breakthrough solutions, mentor top talent, and drive technical excellence across our AI initiatives. You'll work on cutting-edge problems that directly impact how enterprises worldwide leverage data.

What You'll Bring

• ML/LLM: Led fine-tuning (SFT, DPO, RLHF), distillation, quantization with PyTorch/HuggingFace
• Production AI agents: Shipped with LangChain, LlamaIndex, CrewAI
• RAG systems: Expert in vector DBs and prompt orchestration

Leadership Excellence

• Code weekly prototypes, debug complex issues, and unblock your team
• Mentor staff engineers while remaining the go-to expert on the hardest problems
• Partner with PMs and executives on strategy, metrics, and trade-offs

Proven Track Record

• Led 2+ open-ended, user-facing AI projects from ambiguity to impact
• Owned vision, roadmap, and MVPs for AI initiatives
• Presented technical strategies to the C-suite and board
• Enforced MLOps/DevOps excellence: CI/CD, observability, A/B testing

Agentic AI Specialization

• Built multi-agent systems (ETL automation, code-gen loops)
• Created evaluation harnesses and benchmarking systems
• Delivered reliability and latency improvements at scale

Cloud Architecture

• Architected systems handling 10,000+ RPS (AWS/GCP, Kubernetes, serverless)
• Fluent in Python, Go, Java, Scala
• Expert in APIs, async systems, and performance optimization

What Makes You Stand Out

• Experience with code generation (Codex, text-to-SQL) or semantic extraction
• Knowledge graphs expertise (Neo4j, Neptune)
• Big data engine experience (Apache Spark)
• Compiler development (SQL, Python, Scala)
• ML model optimization for low-latency, high-throughput production
• Open-source AI/ML contributions (Hugging Face, PyTorch)
• Retrieval systems or vector database expertise (Pinecone, Weaviate)
• Strategic tech evaluation and build vs. buy decision-making

Position Details

Experience Level: 7+ years in industry (Mid to Staff Level)

Compensation $250,000 - $350,000 base salary

Location: San Francisco, California (Hybrid)

Note: Compensation reflects base pay only. The total package includes equity and comprehensive benefits. Pay varies based on achievements, skills, experience, and location.

Benefits & Perks

• 99% employee health insurance coverage, 75% for dependents
• $200/month stipend for remote work, wellness, gym, and self-care
• Flexible PTO to recharge when you need it
• Professional development allowance for continuous learning
• Company-sponsored Long Term Disability and Life Insurance
• FSA/HSA, dental, vision, and comprehensive health benefits
• Direct impact on an innovative, industry-leading platform
• End-to-end ownership of transformative projects

£2,000 signing bonus paid to you on hire.

Benefits and perks may vary by country.

Requirements

• ML/LLM: Led fine-tuning (SFT, DPO, RLHF), distillation, quantization with PyTorch/HuggingFace
• Production AI agents: Shipped with LangChain, LlamaIndex, CrewAI
• RAG systems: Expert in vector DBs and prompt orchestration

Tech stack

• CI/CD
• Kubernetes
• Python
• Machine Learning
• AWS / GCP
• neo4j
• PyTorch
• Scala
• SQL
• Serverless
• Go
• Java
• Spark