Senior Data Engineer-DataBricks

Job Summary

• Advanced SQL
• Python or Scala for data engineering
• ETL pipeline development
• Cloud platforms (AWS/GCP/Azure)
• Azure 1st party services (ADF, Azure Databricks, Synapse, etc.)
• Big data tools (Spark, Hadoop)
• Data warehousing (Redshift, Snowflake, Big Query)
• Workflow orchestration tools (Airflow, Prefect, or similar)
  

• Design, develop, and maintain scalable and reliable data pipelines
• Demonstrate experience and leadership across two full project cycles using Azure Data Factory, Azure Databricks, and PySpark
• Collaborate with data analysts, scientists, and software engineers to understand data needs
• Implement data quality checks and monitoring systems
• Optimize data delivery and processing across a wide range of sources and formats
• Ensure security and governance policies are followed in all data handling processes
• Evaluate and recommend tools and technologies to improve data engineering capabilities
• Lead and mentor junior data engineers as needed
• Work with cross-functional teams in a dynamic and fast-paced environment
  

Qualifications

• Bachelor’s or Master’s degree in Computer Science, Engineering, or related field
• Certification in cloud platforms or big data technologies (preferred)
• 5+ years of data engineering experience
  

• Programming: Python, Scala, SQL
• Big Data: Spark, Hadoop, Hive
• Stream Processing: Storm, Spark Streaming
• Data Warehousing: Snowflake, Big Query, Redshift
• Cloud: AWS (S3, Lambda, Glue), GCP, Azure (ADF, Azure Databricks)
• Orchestration: Apache Airflow, Prefect, Luigi
• Databases: PostgreSQL, MySQL, NoSQL (MongoDB, Cassandra)
• Tools: Git, Docker, Kubernetes (basic), CI/CD
  

• Strong problem-solving and analytical thinking
• Excellent verbal and written communication
• Ability to manage multiple tasks and deadlines
• Collaborative mindset with a proactive attitude
• Strong analytic skills related to working with unstructured datasets
  

• Experience with real-time data processing (Kafka, Flink)
• Knowledge of data governance and privacy regulations (GDPR, HIPAA)
• Familiarity with ML model data pipeline integration
  

• Minimum 5 years of relevant experience in data engineering roles
• Experience with Azure 1st party services across at least two full project lifecycles
  

Compensation & Benefits

• Competitive salary and annual performance-based bonuses
• Comprehensive health and optional Parental insurance.
• Retirement savings plans and tax savings plans.
• Work-Life Balance: Flexible work hours
  

• Timely development and delivery of high-quality data pipelines
• Implementation of scalable data architectures
• Collaboration with cross-functional teams for data initiatives
• Compliance with data security and governance standards
  

• Uptime and performance of data pipelines
• Reduction in data processing time
• Number of critical bugs post-deployment
• Stakeholder satisfaction scores
• Successful data integrations and migrations