Data Engineer

Data Engineer

Roles and Responsibilities:

• Design, develop, and maintain data pipelines and ETL processes
• Ensure data quality and reliability across various data sources and systems
• Collaborate with cross-functional teams to understand data requirements and deliver solutions
• Analyze and model client, market, and key performance data to generate business insights
• Work on data integration and migration projects ensuring efficient data flow
• Develop and optimize BigQuery SQL scripts for performance tuning
• Utilize cloud-native services such as Google Cloud Storage, BigQuery, Cloud Functions, Pub/Sub, Composer, and Kubernetes
• Automate CI/CD pipelines using tools like Azure DevOps, GitHub, JIRA, and Confluence
• Write clean, efficient Python code using libraries such as Pandas and NumPy
• Troubleshoot and resolve data pipeline issues independently or as part of a team
• Work effectively in an Agile development environment
• Mentor and guide junior team members when needed

Professional and Technical Skills:

• Expertise in SQL and Python programming (No Flex)
• Strong hands-on experience with Google BigQuery and BigQuery SQL performance tuning
• Proficiency with cloud-native data platform services on GCP (Google Cloud Platform)
• Experience with Shell scripting, Oracle, and various data structures (dictionary, array, list, tree)
• Knowledge of automated CI/CD pipelines and tools (Control-M, GitHub, Azure DevOps)
• Familiarity with Agile methodologies and collaborative teamwork
• GCP certification preferred

Professional Attributes:

• Good communication skills
• Strong analytical and problem-solving abilities
• Ability to work independently and in a team environment
• Capability to manage and handle teams