We are seeking a highly skilled Data Platform Infrastructure Engineer to design, build, and manage scalable data platforms across on-premise and cloud environments . The role involves working with cluster technologies , infrastructure automation, and modern data ecosystems to enable reliable and high-performing data platforms.
Key Responsibilities
* Design, deploy, and manage data platform infrastructure across on-prem (Cloudera) and cloud (AWS, Databricks) environments
* Build and maintain distributed data clusters ensuring high availability, scalability, and performance
* Automate infrastructure provisioning using Terraform and Ansible
* Manage and optimize Cloudera Hadoop ecosystems (HDFS, Hive, Spark, YARN, etc.)
* Deploy and manage Databricks workspaces , clusters, and integrations on AWS
* Implement infrastructure-as-code (IaC) and configuration management best practices
* Monitor cluster performance, troubleshoot issues, and ensure system reliability
* Collaborate with data engineers, architects, and DevOps teams to support data pipelines and analytics workloads
* Ensure security, compliance, and governance across data platforms
* Support migration from on-prem to cloud-based data platforms
Technical Skills Required
Core Technologies
* Strong experience in Cloudera (CDH/CDP) cluster setup and administration
* Hands-on experience with Databricks (cluster management, jobs, notebooks)
* Strong exposure to AWS (EC2, S3, IAM, VPC, EMR, networking concepts)
Infrastructure & Automation
* Expertise in Terraform (mandatory) for infrastructure provisioning
* Proficiency in Ansible for configuration management and automation
* Experience with CI/CD pipelines for infrastructure deployments
* Strong understanding of:
* Spark performance tuning and cluster optimization
* Knowledge of containerization (Docker/Kubernetes) is a plus
#J-18808-Ljbffr
Key Responsibilities
* Design, deploy, and manage data platform infrastructure across on-prem (Cloudera) and cloud (AWS, Databricks) environments
* Build and maintain distributed data clusters ensuring high availability, scalability, and performance
* Automate infrastructure provisioning using Terraform and Ansible
* Manage and optimize Cloudera Hadoop ecosystems (HDFS, Hive, Spark, YARN, etc.)
* Deploy and manage Databricks workspaces , clusters, and integrations on AWS
* Implement infrastructure-as-code (IaC) and configuration management best practices
* Monitor cluster performance, troubleshoot issues, and ensure system reliability
* Collaborate with data engineers, architects, and DevOps teams to support data pipelines and analytics workloads
* Ensure security, compliance, and governance across data platforms
* Support migration from on-prem to cloud-based data platforms
Technical Skills Required
Core Technologies
* Strong experience in Cloudera (CDH/CDP) cluster setup and administration
* Hands-on experience with Databricks (cluster management, jobs, notebooks)
* Strong exposure to AWS (EC2, S3, IAM, VPC, EMR, networking concepts)
Infrastructure & Automation
* Expertise in Terraform (mandatory) for infrastructure provisioning
* Proficiency in Ansible for configuration management and automation
* Experience with CI/CD pipelines for infrastructure deployments
* Strong understanding of:
* Spark performance tuning and cluster optimization
* Knowledge of containerization (Docker/Kubernetes) is a plus
#J-18808-Ljbffr
Platform Infrastructure Engineer in scottsdale at Unknown Company
This position is listed as full time and onsite.