Data Engineer

Project Role Description:

• Design, develop, and maintain data solutions for data generation, collection, and processing.
• Create data pipelines and ensure data quality.
• Implement ETL (extract, transform, and load) processes to migrate and deploy data across systems.
• Collaborate with cross-functional teams to understand data requirements and deliver effective solutions.
• Troubleshoot issues in data flow and processing stages.
• Enhance data infrastructure to ensure data accessibility and reliability for decision-making.

Roles & Responsibilities:

• Deliver data integration and data warehousing solutions.
• Develop and optimize SQL queries, particularly in Google BigQuery.
• Work with cloud-native services such as bucket storage, Google BigQuery, Cloud Functions, Pub/Sub, Composer, and Kubernetes.
• Automate CI/CD pipelines using tools like Azure DevOps (AZDO), Control-M, GitHub, JIRA, and Confluence.
• Perform performance tuning of BigQuery SQL scripts.
• Use Python for scripting and data manipulation, leveraging libraries such as Pandas and NumPy.
• Work in an agile environment and adapt quickly to new technologies.
• Collaborate effectively with various teams and work independently when required.
• Participate in shift B work schedule and contribute as an individual contributor.

Professional & Technical Skills:

• Must have skills: Google BigQuery, SQL, Python (including Pandas, NumPy), Shell Scripting
• Good to have skills: Google Cloud Data Services, Microsoft SQL Server, GCP Certification
• Experience with data processing, Pub/Sub, Dataflow, Kafka streaming, Looker
• Strong knowledge of cloud-native data platform services
• Experience in CI/CD pipeline automation and performance tuning

Professional Attributes:

• Excellent communication skills
• Strong analytical and problem-solving abilities
• Ability to collaborate with cross-functional teams
• Capable of working independently and managing tasks with minimal supervision
• Good team handling and leadership skills

Educational Qualification:

• 15 years of full-time education

Additional Information:

• Minimum 5 years of experience required
• Ready to work in shift B schedule