Job
Description
As a Lead Big Data Developer at CGI, you will play a crucial role in designing, building, and optimizing large-scale data pipelines and processing systems. Your expertise in Python, PySpark, and SQL will be instrumental in leading data initiatives, mentoring team members, and delivering high-performance data solutions. Here are your key responsibilities: - Lead the design and development of scalable, efficient, and reliable data pipelines using PySpark, Python, and SQL. - Collaborate with data architects, analysts, and business stakeholders to understand data requirements and translate them into technical solutions. - Optimize data workflows for performance, scalability, and cost efficiency in big data environments. - Implement data ingestion, transformation, and aggregation processes from multiple structured and unstructured sources. - Ensure data quality, integrity, and consistency through validation, testing, and monitoring frameworks. - Design and enforce best practices for coding, version control, and CI/CD within the data engineering team. - Provide technical leadership and mentorship to junior and mid-level developers. - Collaborate with DevOps and DataOps teams for deployment and operationalization of data solutions. - Stay updated with the latest technologies and trends in the big data ecosystem. To be successful in this role, you should have: - 8+ years of experience in data engineering or big data development, with at least 3+ years in a lead or senior role. - Strong proficiency in Python, PySpark, and SQL. - Experience with cloud platforms (AWS, Azure, or GCP) and data orchestration tools. - Solid understanding of data modeling, ETL design, and performance optimization. - Excellent communication and leadership skills. Preferred qualifications include experience with Delta Lake, Apache Iceberg, Hudi, containerization, orchestration, and machine learning pipelines. A certification in AWS Big Data, GCP Data Engineer, or Azure Data Engineer is a plus. Education: - Bachelors or masters degree in computer science, Information Technology, or a related field. In addition to the technical aspects of the role, CGI offers a work environment rooted in ownership, teamwork, respect, and belonging. As a CGI Partner, you will have the opportunity to contribute to the company's strategy and direction from day one, develop innovative solutions, build relationships with teammates and clients, and shape your career with the support of leaders who care about your well-being and growth opportunities. As a Lead Big Data Developer at CGI, you will play a crucial role in designing, building, and optimizing large-scale data pipelines and processing systems. Your expertise in Python, PySpark, and SQL will be instrumental in leading data initiatives, mentoring team members, and delivering high-performance data solutions. Here are your key responsibilities: - Lead the design and development of scalable, efficient, and reliable data pipelines using PySpark, Python, and SQL. - Collaborate with data architects, analysts, and business stakeholders to understand data requirements and translate them into technical solutions. - Optimize data workflows for performance, scalability, and cost efficiency in big data environments. - Implement data ingestion, transformation, and aggregation processes from multiple structured and unstructured sources. - Ensure data quality, integrity, and consistency through validation, testing, and monitoring frameworks. - Design and enforce best practices for coding, version control, and CI/CD within the data engineering team. - Provide technical leadership and mentorship to junior and mid-level developers. - Collaborate with DevOps and DataOps teams for deployment and operationalization of data solutions. - Stay updated with the latest technologies and trends in the big data ecosystem. To be successful in this role, you should have: - 8+ years of experience in data engineering or big data development, with at least 3+ years in a lead or senior role. - Strong proficiency in Python, PySpark, and SQL. - Experience with cloud platforms (AWS, Azure, or GCP) and data orchestration tools. - Solid understanding of data modeling, ETL design, and performance optimization. - Excellent communication and leadership skills. Preferred qualifications include experience with Delta Lake, Apache Iceberg, Hudi, containerization, orchestration, and machine learning pipelines. A certification in AWS Big Data, GCP Data Engineer, or Azure Data Engineer is a plus. Education: - Bachelors or masters degree in computer science, Information Technology, or a related field. In addition to the technical aspects of the role, CGI offers a work environment rooted in ownership, teamwork, respect, and belonging. As a CGI Partner, you will have the opportunity to contribute to the company's strategy and direction from day one, develop innovative solutions, build relationships with teammates and clients, and shape your career wit
