Data EngineerDesign scalable data pipelines on GCP (BigQuery, Dataproc, GCS) Build batch and streaming data workflows within Google Cloud Platform (GCP) leveraging Cloud Pub/Sub and Cloud Dataflow Use Databricks notebooks with PySpark to clean, transform, and prepare large datasets Optimize performance of BigQuery queries and Dataproc Spark jobs Troubleshoot Cloud Composer (Airflow), Dataproc, and BigQuery jobs. Build Glue jobs in PySpark to clean and transform raw data Experience with Spark Streaming, Spark MLlib, Scala and Creating the Data Frames in Spark. Provide data to Appian bots for Gen AI needs, ensure data availability for AI-driven automation and decision-making processes.
GCP ML Data Engineer in atlanta at Unknown Company
This position is listed as full time and onsite.