Opening For Sr Data Engineer - Azure Databricks AWS

Key Responsibilities:

• Lead the design, development, and deployment of advanced data pipelines and analytical workflows leveraging Azure Databricks, PySpark, and other big data technologies.
• Implement and optimize ETL/ELT processes for data ingestion, transformation, and loading from diverse sources into data lakes and data warehouses on Azure and AWS.
• Develop and maintain data models, ensuring data quality, integrity, and governance.
• Utilize Python and SQL extensively for data manipulation, scripting, automation, and complex data analysis.
• Work with AWS services (e.g., S3, Glue, Redshift, Lambda) and Azure services (e.g., Data Lake, Data Factory, Synapse Analytics) to build and manage data infrastructure.
• Monitor, troubleshoot, and optimize data pipeline performance and reliability, ensuring data availability and accuracy.
• Collaborate with data scientists, analysts, and business stakeholders to understand data requirements and translate them into technical solutions.
• Implement robust data validation and testing frameworks to ensure data quality and compliance.
• Mentor junior data engineers and provide technical guidance on best practices in data engineering and cloud technologies.
• Contribute to the implementation of CI/CD pipelines for data solutions and infrastructure as code.

Required Skills & Qualifications:

• Bachelor's degree in Computer Science, Engineering, or a related field.
• Proven experience as a Data Engineer, with a significant portion in a senior or lead role.
• Expertise in Azure Databricks, including Delta Lake and Apache Spark.
• Strong proficiency in Python and SQL for data engineering tasks.
• Hands-on experience with AWS cloud services for data storage, processing, and analytics.
• Solid understanding of data warehousing concepts, dimensional modeling, and ETL/ELT principles.
• Experience with data governance, data quality, and security best practices.
• Familiarity with CI/CD pipelines and DevOps practices for data solutions.
• Excellent problem-solving, analytical, and communication skills.
• Ability to work independently and as part of a collaborative team