Senior Data Engineer - GPM

About the Role:

Senior Data Engineer

Qualifications:

• 5-8 years

  of experience in data engineering, with a focus on building and managing data pipelines.
• Strong proficiency inrelational databases(e.g., MySQL, PostgreSQL) andNoSQL databases(e.g., MongoDB, Cassandra).
• Experience in building data pipelines with data warehouses likeSnowflake,Redshift
• Experience in processing unstructured data stored fromS3usingAthena, Glue etc.
• Hands-on experience withKafkafor real-time data streaming and messaging.
• Solid understanding ofETL processes, data integration, and data pipeline optimization.
• Proficiency in programming languages likePython,Java, orScalafor data processing.
• Experience withApache Sparkfor big data processing and analytics is an advantage
• Familiarity with cloud platforms likeAWS,GCP, orAzurefor data infrastructure is a plus.
• Excellent problem-solving skills and attention to detail.
• Strong communication and collaboration skills, with the ability to work effectively in a team environment.

Key Responsibilities:

• Design, build, and maintain efficient and scalable data pipelines to support data integration and transformation across various sources.
• Work with relational databases (e.g., MySQL, PostgreSQL, etc.) and NoSQL databases (e.g., MongoDB, Cassandra, etc.) to manage and optimize large datasets.
• Utilize Apache Spark for distributed data processing and real-time analytics.
• Implement and manage Kafka for data streaming and real-time data integration between systems.Collaborate with cross-functional teams to gather and translate business requirements into technical solutions.
• Monitor and optimize the performance of data pipelines and architectures, ensuring high availability and reliability.
• Ensure data quality, consistency, and integrity across all systems.
• Stay up-to-date with the latest trends and best practices in data engineering and big data technologies.