AI Software EngineerThe Fitch Group Emerging Technology AI group is seeking an AI Software Engineer to be part of a team that will be dedicated to build and support Generative AI, Machine learning, Deep Learning and Data science solutions across the organization. The position could be based out of our Chicago or NY offices. Fitch is building a unified platform that supports both internal Centers of Excellence (CoE) Python-based agent frameworks for complex, pro-code use cases and no-code/low-code solutions such as Microsoft Copilot Studio and UiPath.
This platform will centralize governance, evaluation tooling, and LLMOps capabilities so teams across Fitch can confidently design, deploy, and operate AI agents. We're seeking a contractor AI Software Engineer with deep Python and cloud computing experience to design, implement, and operationalize this platform end-to-end.What We Offer:This will be a high impact role with significant visibility where the candidate will work on some flagship Fitch productsThe candidate will have an excellent opportunity to work in the cutting-edge field of Generative AI and LLMOpsFitch promotes an excellent work culture and is known for providing a good work life balanceWe'll Count on You To:Platform architecture and developmentDesign and build a modular platform that integrates Python-based agent frameworks (e.g., LangChain, Semantic Kernel, custom orchestration) and no-code/low-code tools (e.g., MS Copilot Studio).Implement core services: agent orchestration, prompt/policy management, connector/tool registry, configuration store, and secure secrets handling.Develop APIs/SDKs/adapters to support multiple model providers and runtimes; enable multi-tenant use by various teams.LLMOps, evaluation, and observabilityStand up experiment tracking and evaluation pipelines (offline/online, human-in-the-loop, rubric/golden set scoring).Build telemetry for prompt usage, latency, cost, safety incidents, grounding/accuracy metrics, and drift detection.Enable A/B testing, canary releases, and rollback strategies for agents and prompts.Governance, security, and complianceEmbed policy enforcement for PII handling, data residency, model/provider allowlists, and approval workflows.Implement role-based access control (RBAC), audit logging, and change management for agent lifecycle.Support model risk management artifacts (model cards, decision logs, evaluation reports).Reliability, CI/CD, and infrastructureDevelop secure-by-design services: least-privilege access, secrets rotation, content filtering, isolation of external tools.Build robust CI/CD, automated tests, IaC, and environment promotion; ensure high availability and resilience.Integrations and data groundingCreate connectors to enterprise systems (APIs, data lakes, knowledge bases) and retrieval pipelines (RAG) with vector search.Integrate with Microsoft ecosystem (Copilot Studio) while leveraging AWS-first infrastructure and services.Collaboration and enablementPartner with CoEs, security, compliance, and product teams to translate requirements into platform features.Produce documentation, usage guides, templates, and reference implementations for pro-code and no-code users.Support onboarding, troubleshoot issues, and drive adoption across Fitch teams.What You Need to Have:5+ years of software engineering experience building production-grade platforms or distributed systemsExpert-level Python skills, including building services, SDKs, and integrationsStrong AWS experience container orchestration (EKS/Fargate), serverless (Lambda), API Gateway, Step Functions, S3, DynamoDB/RDS, Secrets Manager/KMS, CloudWatch/CloudTrail, IAMHands-on with LLM/agent frameworks and providers (e.g., LangChain, Semantic Kernel, OpenAI/AWS Bedrock/Anthropic SDKs) and prompt/agent lifecycle managementProven LLMOps experience: experiment tracking, offline/online evaluation, telemetry, cost/safety monitoring, and A/B testingSecurity and governance expertise: RBAC/ABAC, audit logging, policy enforcement, PII controls, approvals workflowsData retrieval/RAG proficiency: vector databases (e.g., OpenSearch vector, pgvector, Pinecone), embedding pipelines, grounding strategiesCI/CD and IaC: GitHub Actions/AWS CodePipeline, Terraform/CloudFormation; testing frameworks and secure supply chain practicesStrong communication skills; ability to collaborate with cross-functional stakeholdersWhat Would Make You Stand Out:Experience integrating Microsoft Copilot Studio, or Power Platform into enterprise ecosystemsFamiliarity with evaluation methods for LLMs (assertion tests, rubric scoring, golden datasets, red-teaming, human review loops)Observability stack experience (OpenTelemetry, Prometheus/Grafana) and cost/usage analyticsBackground in model risk management or regulated environmentsKnowledge of multi-tenant architecture and guardrails for internal platforms