Senior Data Engineer

Role Overview: As a Senior Data Engineer, your primary responsibility is to design and optimize data pipelines, ETL workflows, and cloud-based data solutions using your expertise in Databricks, Apache Airflow, SQL, and Python. You play a crucial role in driving efficient and scalable data engineering solutions by implementing big data processing techniques and automation. Key Responsibilities: - Design, develop, and optimize data pipelines utilizing Databricks and Apache Airflow. - Implement PySpark-based transformations and processing in Databricks for handling large-scale data efficiently. - Develop and maintain SQL-based data pipelines, focusing on performance tuning and optimization. - Create Python scripts for automation, data transformation, and API-based data ingestion. - Work with Airflow Directed Acyclic Graphs (DAGs) to schedule and orchestrate data workflows effectively. - Optimize data lake and data warehouse performance to ensure scalability and reliability. - Integrate data pipelines with cloud platforms such as AWS, Azure, or GCP, and various data storage solutions. - Ensure strict adherence to data security, governance, and compliance standards. Qualifications Required: - 8-9 years of experience in Data Engineering or related fields. - Strong expertise in Databricks including PySpark, Delta Lake, and DBSQL. - Proficiency in Apache Airflow for scheduling and orchestrating workflows. - Advanced SQL skills for data extraction, transformation, and performance tuning. - Strong programming skills in Python covering pandas, NumPy, PySpark, and APIs. - Experience with big data technologies and distributed computing. - Hands-on experience with cloud platforms like AWS, Azure, or GCP. - Expertise in data warehousing and data modeling concepts. - Understanding of CI/CD pipelines and version control using Git. - Experience with data governance, security, and compliance best practices. - Excellent troubleshooting, debugging, and performance optimization skills.,