Job Overview
A global consulting and technology services organization is seeking a Big Data / Spark Principal Architect to lead the design and implementation of large-scale data systems. This senior-level role combines deep technical expertise in Spark, Databricks, and cloud-based data platforms with a strategic understanding of solutioning and consulting across enterprise environments. The successful candidate will architect high-performance data pipelines, guide technical teams, and act as a subject matter expert (SME) for big data and streaming solutions within the Media & Entertainment domain.
Key Responsibilities
- Architecture & Design: Define and implement scalable, high-performance big data architectures using Spark (DataFrame, Spark SQL, PySpark/Scala Spark) and Databricks.
- Solutioning & Consulting: Lead solution design engagements, act as SME for big data technologies, and provide consulting expertise for enterprise data strategies.
- Pipeline Development: Build and optimize batch and streaming data pipelines leveraging Spark, AWS Glue, and Kafka (or equivalent streaming tools).
- Data Governance & Quality: Establish frameworks for data quality, lineage, and compliance across multi-source data ecosystems.
- Cloud Enablement: Architect and deploy solutions on AWS or Azure, ensuring performance, scalability, and cost efficiency.
- Practice Development: Contribute to the growth of the Big Data practice through mentoring, pre-sales support, and best practice documentation.
- Collaboration: Work closely with data engineers, analysts, and business stakeholders to translate requirements into robust technical solutions.
- Continuous Improvement: Stay current with emerging technologies and promote adoption of modern data engineering practices.
Must-Have Requirements
- 10–18 years of total experience in Big Data Architecture, Engineering, and Consulting.
- Strong hands-on expertise in Apache Spark (DataFrame, SQL, PySpark/Scala Spark) and Databricks.
- Proven track record in solutioning and consulting as a Subject Matter Expert (SME) in Big Data.
- Experience with AWS Glue or Azure Data Services for data ingestion and transformation.
- Familiarity with streaming technologies such as Kafka, Kinesis, or Flink.
- Deep understanding of ETL design, data governance, and data quality frameworks.
- Strong coding skills and willingness to be hands-on with development.
- Prior experience in Media & Entertainment or related domains.
- Must be a U.S. Citizen or Green Card Holder (mandatory).
- Demonstrated job stability and long-term project ownership.
Nice-to-Have Skills
- Exposure to AI/ML integration within big data ecosystems.
- Experience leading practice development or solution selling for enterprise clients.
- Knowledge of modern data mesh or lakehouse architectures.
Location: USA – Remote (Full-Time)
Work Type: Permanent Remote
Experience Required: 10 – 18 Years
Compensation: USD 150,000 – 200,000 (plus bonus, based on experience and performance)
#J-18808-Ljbffr