Job
Description
DevOn is a leading provider of innovative technology solutions focusing on data-driven decision-making, cloud computing, and advanced analytics. We are passionate about solving complex business problems through technology and currently seeking a skilled and motivated Data Engineer Lead to join our dynamic team. As a Data Engineer Lead at DevOn, your primary responsibility will be to lead the design, development, and maintenance of data pipelines and ETL workflows, utilizing modern cloud technologies. You will collaborate closely with cross-functional teams to ensure data availability, reliability, and scalability, facilitating data-driven decision-making throughout the organization. This role necessitates a profound understanding of Python, PySpark, AWS Glue, RedShift, SQL, Jenkins, Bitbucket, EKS, and Airflow. Key Responsibilities: - Lead the design and implementation of scalable data pipelines and ETL workflows in a cloud environment, primarily AWS. - Develop and manage data ingestion, transformation, and storage frameworks using AWS Glue, PySpark, and RedShift. - Architect and enhance complex SQL queries for large datasets, ensuring data integrity across systems. - Work collaboratively with data scientists, analysts, and business stakeholders to comprehend data requirements and deliver high-quality data solutions. - Automate the end-to-end data pipeline process using Jenkins and Bitbucket, promoting efficient CI/CD practices. - Manage and optimize data orchestration utilizing Apache Airflow. - Provide technical leadership and mentorship to junior team members, ensuring adherence to best practices in data engineering. - Utilize AWS services such as RedShift, S3, Lambda, and EKS for deploying and managing data solutions. - Resolve complex data pipeline issues promptly, ensuring minimal downtime and high availability. - Participate in architecture and design reviews, contributing insights on technical solutions and enhancements. - Continuously assess new tools and technologies to enhance the efficiency and scalability of our data infrastructure. Required Skills and Qualifications: - 5+ years of professional experience in Data Engineering, demonstrating expertise in building scalable data pipelines and ETL workflows. - Proficiency in Python for data processing and scripting. - Hands-on experience with PySpark for large-scale data processing. - Thorough knowledge of AWS Glue, RedShift, S3, and other AWS services. - Advanced skills in SQL for data manipulation and optimization. - Experience with Jenkins and Bitbucket for CI/CD automation. - Familiarity with EKS (Elastic Kubernetes Service) for containerized deployment of data applications. - Proficiency in Apache Airflow for data orchestration and workflow automation. - Strong problem-solving abilities and adeptness in debugging complex data workflow issues. - Excellent communication skills to collaborate effectively with cross-functional teams and articulate technical concepts clearly. - Ability to work in an Agile development environment, managing multiple priorities and meeting tight deadlines. Preferred Qualifications: - Experience with additional AWS services (e.g., Lambda, Redshift Spectrum, Athena). - Familiarity with Docker and container orchestration technologies like Kubernetes. - Knowledge of data modeling and data warehousing concepts. - Bachelor's or Master's degree in Computer Science, Engineering, or a related field.,
