Job
Description
JD - Data Engineer (4+) Key Responsibilities: Design, build, and maintain scalable data pipelines using Apache Spark on Databricks. Integrate data from various structured and unstructured sources into the data lake/data warehouse. Collaborate with data scientists, analysts, and business stakeholders to understand data requirements. Develop and maintain ETL/ELT jobs using PySpark, SQL, or Scala. Ensure high data quality and data governance across pipelines and platforms. Optimize performance of large-scale data processing jobs. Implement best practices in data engineering, including version control, code reviews, and automation. Work with DevOps/Cloud teams to deploy pipelines in production on cloud platforms (e.g., Azure Databricks, AWS, or GCP). Required Skills: 4 + years of experience as a Data Engineer. Strong expertise in Databricks, Apache Spark, and PySpark. Proficiency in SQL and working with large datasets. Experience in one or more programming languages: Python, Scala, or Java. Knowledge of Delta Lake and data lakehouse architecture. Familiarity with data modeling and data warehousing concepts. Hands-on experience with cloud platforms (Azure, AWS, or GCP). Experience with CI/CD pipelines and version control tools (e.g., Git). Preferred/Bonus Skills: Experience with Azure Data Factory, AWS Glue, or other ETL tools. Knowledge of streaming data technologies (Kafka, Spark Streaming). Understanding of data governance, privacy, and compliance (e.g., GDPR, HIPAA). Familiarity with ML model pipelines and MLflow in Databricks is a plus. Data Engineer - Aws Azure Cloud
