Job TitleRequired:Strong verbal and written communication skills to effectively articulate messages to internal and external teams.Hands on experience with Azure data platform stack (Azure Databricks, Azure Data Factory, Azure Data Lake storage, Azure DevOps, Azure Functions)Experience with dimensional modeling and data warehousing conceptsExperience with designing, building, optimizing, troubleshooting end-to-end big data pipelines using structured (relational and files) and semi-structured dataExperience with building metadata driven data processing frameworksStrong experience in Spark, SQL, python, Pyspark, shell scripting.Ability to work in an agile team.Ability to take ownership of a request from initial requirements, design, development and production deploymentNice to have:Powershell, scala, javaApache AirflowAzure EventHubsApace KafkaStreaming dataCosmosDB/NoSQL DatabaseRequired Skills: Azure, Windows PowerShell, Python, SQL Basic Qualification: Requirements: Hands on experience with Azure data platform stack (Azure Databricks, Azure Data Factory, Azure Data Lake storage, Azure DevOps) Experience with designing, building and productionizing big data pipelines Strong experience in SQL, python, pyspark, shell scripting is required. Must have worked with processing and optimizing multi-billion record tables and tables with 100's gigs in size Ability to build pipelines to process structured and semi-structured data. Ability to work with in an agile team Ability to take ownership of a feature request from initial requirements, design, development and production deployment Ability to build processes that support data transformation, workload management, data structures, dependency and metadata Additional preferred but not required: Powershell, scala, java Apache Airflow Azure EventHubs Streaming data Object Oriented Programming Required: Experience working with SQL, python, pyspark, shell scripting Understands concepts of dimensional data modeling and experience with loading a dimensional data warehouse Ability to work with in an agile team Experience with writing unit tests and packaging code Ability to take ownership of a feature request from initial requirements, design, development and production deployment Ability to build processes that support data transformation, workload management, data structures, dependency and metadata Ability to build pipelines to process structured and semi-structured data.
Ability to translate mapping/SQL logic from specifications to pyspark deployable code Ability to troubleshoot issues and provide solution recommendations Experience with designing, building and productionizing big data pipelines