The Lead Cloudera Consultant (Solution Architect) will design, build, and optimize real-time streaming pipelines using the full Cloudera Data Platform (CDP) streaming stack.
This role requires deep hands-on expertise with NiFi, Kafka, Flink, Kudu/Impala, and SQL Stream Builder (SSB) to deliver mission-critical, high-volume Data-in-Motion solutions.
The consultant will architect and implement real-time ingestion, transformation, CDC patterns, and integration pipelines, while guiding engineering teams on best practices across the streaming ecosystem.
Key Responsibilities
- Architect and build end-to-end real-time pipelines using NiFi, Kafka, Flink, Kudu/Impala, and SSB.
- Own architectural decisions, design patterns, schema evolution strategies, state management, and exactly-once delivery.
- Develop advanced NiFi flows including JDBC/DBCP services, record processors, stateful processors, schema registry integration, and high-volume ingestion patterns.
- Build and optimize Flink SQL and DataStream API jobs including event-time windows, watermarks, checkpointing, and state backends.
- Design and tune Kudu tables including primary keys, partition strategies, distribution, upserts, deletes, and merges.
- Build, deploy, and monitor streaming SQL jobs using SQL Stream Builder (SSB).
- Deliver core real-time use cases including NiFi?Snowflake?Impala/Kudu ingestion, Kafka?Flink streaming, Flink?Kafka with exactly-once semantics, and CDC ingestion.
- Tune NiFi, Kafka, and Flink clusters to optimize throughput, reliability, and latency.
- Implement schema governance, back-pressure controls, error-handling frameworks, and replay strategies.
- Collaborate with platform engineering teams to optimize deployments and configurations.
- Provide architectural guidance, documentation, and mentorship to engineering teams.
Required Qualifications
- 8+ years of experience in data engineering or streaming system development.
- 3–5+ years of hands-on experience with Cloudera CDP/CDF streaming components.
- Strong production-grade experience with NiFi, including advanced processors, controller services, CDC patterns, and Snowflake/Kudu ingestion.
- Hands-on expertise with Kafka including topic design, consumer groups, schema registry, partition strategies, and high-volume streaming.
- Strong proficiency with Flink SQL and DataStream API including event-time processing, windows, watermarks, checkpointing, and exactly-once semantics.
- Experience with Kudu table design, schema changes, upserts, deletes, merges, and Impala querying.
- Strong knowledge of SQL Stream Builder, job creation, connectors, and Flink SQL job deployment in CDP.
- Experience with CDC ingestion using NiFi, Flink CDC, or SSB including late events, schema evolution, deletes, and incremental key management.
- Strong SQL and distributed systems fundamentals.
- Experience in high-volume industries such as financial services, healthcare, or telecom is preferred.
Preferred Qualifications
- Experience running NiFi, Kafka, or Flink on Kubernetes.
- Snowflake ingestion experience including staging and COPY INTO patterns.
- Experience with Debezium for CDC.
- Experience with CI/CD for data pipelines.
- Security knowledge including Kerberos, Ranger, and Atlas.