Splunk Infrastructure EngineerWe are seeking an experienced Splunk Infrastructure Engineer to support a large-scale federal customer environment processing approximately 2.5TB of data per day. This role will focus on maintaining, optimizing and modernizing a complex Splunk deployment that spans multiple domains and supports critical security operations.The ideal candidate will possess deep expertise in Splunk Enterprise architecture, distributed deployments, Search Head Clustering, indexer administration, configuration management, and infrastructure modernization efforts. While the immediate focus is operational support and stability, this position will play a key role in the long-term transformation of the Splunk environment toward a more scalable and cloud-ready architecture.Responsibilities:Maintain and optimize a large-scale VMware-hosted Splunk deployment.Administer Search Head Clusters supporting Enterprise Security and ad-hoc search environments.Manage Splunk indexers, monitoring consoles, license servers, and supporting infrastructure components.Support complex data routing requirements across multiple security domains, including relay and IRDA tiers.Utilize btool to identify, troubleshoot, and remediate configuration inconsistencies across the environment.Manage and improve configuration control processes through Git repositories and version management practices.Support the ongoing implementation and integration of Cribl data pipeline technologies.Assist with migration efforts to transition Splunk infrastructure from legacy domains into modernized environments.Monitor and maintain SmartStore integrations utilizing Pure Storage S3 backends.Optimize performance across multi-homed indexer configurations.Perform system health monitoring, troubleshooting, capacity planning, and performance tuning.Develop and maintain technical documentation, architecture diagrams, and operational procedures.Collaborate with customer stakeholders to support modernization initiatives and future-state architecture planning.Required Qualifications:Active Top Secret clearanceU.S.
Citizenship requiredSplunk Core Consultant Certification (Required)Experience supporting large-scale Splunk environments processing terabytes of daily ingestStrong expertise with:Distributed Splunk architecturesSearch Head ClusteringIndexer administration and managementSplunk Enterprise Security (ES)Splunk configuration management and troubleshootingAdvanced experience utilizing btool for configuration auditing and issue resolutionExperience supporting VMware-based Splunk deploymentsExperience with Git-based configuration management and version controlPreferred Qualifications:Splunk Enterprise Security CertificationExperience with Cribl deployment and migration activitiesExperience supporting SmartStore environmentsExperience integrating Splunk with S3-compatible storage solutionsExperience supporting cloud migration or infrastructure modernization initiativesExperience supporting multi-domain federal environmentsTechnical Environment:Splunk EnterpriseSplunk Enterprise Security (ES)Search Head ClusteringDistributed Indexer ClustersVMwareCriblGitSmartStorePure Storage S3Multi-Domain ArchitecturesMonitoring ConsoleLicense ServerLinux Administration