Sr. Data Engineer - Snowflake

Embark on an exciting journey into the realm of software development with 3Pillar! We extend an invitation for you to join our team and gear up for a thrilling adventure. At 3Pillar, our focus is on leveraging cutting-edge technologies that revolutionize industries by enabling data driven decision making.

As a Senior Data Engineer, you will hold a crucial position within our dynamic team, actively contributing to thrilling projects that reshape data analytics for our clients, providing them with a competitive advantage in their respective industries.

If your passion for data engineering solutions that make a real-world impact, consider this your pass to the captivating world of Data Science and Engineering!

Responsibilities:

• Lead the design and implementation of scalable ETL/data pipelines using Python and Luigi for data processing.
• Ensure efficient data processing for high-volume clickstream, demographics, and business data.
• Guide the team in adopting best practices for data pipeline development, code quality, and performance optimization.
• Configure, deploy, and maintain AWS infrastructure, primarily AWS EC2, S3, RDS, and EMR, to ensure scalability, availability, and security.
• Support data storage and retrieval workflows using S3 and SQL-based storage solutions.
• Provide architectural guidance for cloud-native data solutions and infrastructure design.
• Oversee legacy framework maintenance, identify improvement areas, and propose comprehensive cloud migration or modernization plans.
• Lead the strategic planning, execution, and optimization of large-scale data migration initiatives to Snowflake, ensuring data integrity, security, and minimal business disruption.
• Coordinate infrastructure changes with stakeholders to align with business needs and budget constraints.
• Develop and implement robust monitoring solutions to track system health, performance, and data pipeline accuracy.
• Set up alerts and dashboards for proactive issue detection and collaborate with cross-functional teams to resolve critical issues.
• Lead efforts in incident response, root cause analysis, and post-mortem processes for complex data engineering challenges.
• Document workflows, troubleshooting procedures, and code for system transparency and continuity.
• Provide mentoring and training to team members on best practices and technical skills.
• Foster a culture of continuous learning, knowledge sharing, and technical excellence within the data engineering team.

Minimum Qualifications:

• Experience: 6 years of experience in data engineering or a related technical field, with at least 4 years in a Snowflake projects.

• Cloud Data Warehousing:

  Proven expertise with Snowflake data warehousing, including schema design, efficient data loading (e.g., Snowpipe, COPY into), performance tuning, and robust access control mechanisms.

• Programming & Scripting:

  Strong programming skills in Python for automation and data workflows.

• Data Processing & Storage:

  Expertise in managing SQL databases for data storage and query optimization.

• Monitoring & Alerting Tools:

  Familiarity with monitoring solutions for real-time tracking and troubleshooting of data pipelines.
• Experience with the AWS Cloud and data engineering services

Thank you,

Kiran Dhanak

3Pillar