Senior Data Engineer (Snowflake)

Responsibilities:

• Lead the design and implementation of scalable ETL/data pipelines using Python and Luigi for data processing.
• Ensure efficient data processing for high-volume clickstream, demographics, and business data.
• Guide the team in adopting best practices for data pipeline development, code quality, and performance optimization.
• Configure, deploy, and maintain AWS infrastructure, primarily AWS EC2, S3, RDS, and EMR, to ensure scalability, availability, and security.
• Support data storage and retrieval workflows using S3 and SQL-based storage solutions.
• Provide architectural guidance for cloud-native data solutions and infrastructure design.
• Oversee legacy framework maintenance, identify improvement areas, and propose comprehensive cloud migration or modernization plans.
• Lead the strategic planning, execution, and optimization of large-scale data migration initiatives to Snowflake, ensuring data integrity, security, and minimal business disruption.
• Coordinate infrastructure changes with stakeholders to align with business needs and budget constraints.
• Develop and implement robust monitoring solutions to track system health, performance, and data pipeline accuracy.
• Set up alerts and dashboards for proactive issue detection and collaborate with cross-functional teams to resolve critical issues.
• Lead efforts in incident response, root cause analysis, and post-mortem processes for complex data engineering challenges.
• Document workflows, troubleshooting procedures, and code for system transparency and continuity.
• Provide mentoring and training to team members on best practices and technical skills.
• Foster a culture of continuous learning, knowledge sharing, and technical excellence within the data engineering team.
  

Qualification

• Experience: 6 years of experience in data engineering or a related technical field, with at least 4 years in a Snowflake projects.
• Cloud Data Warehousing: Proven expertise with Snowflake data warehousing, including schema design, efficient data loading (e.g., Snowpipe, COPY into), performance tuning, and robust access control mechanisms.
• Programming & Scripting: Strong programming skills in Python for automation and data workflows.
• Data Processing & Storage: Expertise in managing SQL databases for data storage and query optimization.
• Monitoring & Alerting Tools: Familiarity with monitoring solutions for real-time tracking and troubleshooting of data pipelines.
• Experience with the AWS Cloud and data engineering services