Data Engineer - GSR5/GSR6
This role requires a combination of software engineer, data enginner and an ML developer where youre expected to build solutions integrating data pipelines & ML models. It also requires you to have basic knowledge and (preferably) hands-on experience on building back-end services(APIs) which integrate data & applications.
Required Skills and Qualifications:
- 4+ years of professional experience in software development.
- Strong proficiency and hands-on experience in both
Python(Must-have)
and Java(Nice to have)
. - Experience building and maintaining data pipelines (batch or streaming) preferably on Cloud platforms(especially GCP).
- Experience with at least one major distributed data processing framework (e.g., DBT, DataForm, Apache Spark, Apache Flink, or similar).
- Experience with workflow orchestration tools (e.g., Apache Airflow, Qlik replicate etc).
- Experience working with relational databases (SQL) and understanding of data modeling principles.
- Experience with cloud platforms (Preferably GCP. AWS or Azure will also do) and relevant data services (e.g., BigQuery, GCS, Data Factory, Dataproc, Dataflow, S3, EMR, Glue etc.).
- Experience with data warehousing concepts and platforms (BigQuery, Snowflake, Redshift etc.).
- Understanding of concepts related to integrating or deploying machine learning models into production systems.
- Experience working in an Agile development environment & hands-on in any Agile work management tool(Rally, JIRA etc.).
- Experience with version control systems, particularly Git.
- Solid problem-solving, debugging, and analytical skills.
- Excellent communication and collaboration skills.
- Experience working in a production support team (L2/L3) for operational support.
Preferred Skills and Qualifications (Nice to Have):
- Familiarity with data quality and data governance concepts.
- Experience building and consuming APIs (REST, gRPC) related to data or model serving.
- Bachelors or Masters degree in Computer Science, Engineering, Data Science, or a related field.
Key Responsibilities:
-
Data Pipeline Development:
Design, develop, and maintain scalable and reliable batch data pipelines using Python and Java, leveraging industry standard frameworks like Data proc and Dataflow. -
ELT Implementation:
Implement efficient data extraction, transformation, and loading processes to move data from various sources into data warehouses, data lakes, or other storage systems. -
Backend Development:
Develop and maintain backend services (potentially in Python or Java) that interact with data stores, trigger pipelines, and/or serve data via APIs. -
Data Storage Interaction:
Work with various data storage technologies, including relational databases (SQL), data lakes (GCS, Big Query), and data warehouses (BigQuery). -
Performance Optimization:
Analyze and optimize the performance of data pipelines and related services to ensure efficiency and cost-effectiveness when dealing with large datasets. -
Monitoring and Reliability:
Implement monitoring, logging, and alerting for data pipelines and services to ensure their health, reliability, and data quality. Troubleshoot and resolve production issues. -
Collaboration:
Collaborate effectively with Product Owners, Data Scientists, Data Engineers, MLOps Engineers, and other engineering teams to understand requirements and deliver integrated solutions. -
Code Quality & Testing:
Write clean, well-tested, and maintainable code. Participate in code reviews. -
Technical Contribution:
Contribute to architectural discussions and help define technical approaches for data and AI-related projects. -
AI/ML Integration (Additional responsibility):
Work closely with Data Scientists to operationalize machine learning models. This includes building the infrastructure and code (incl. chatbots) to integrate models into data pipelines or backend services for training data preparation, inference, or prediction serving. -
Operational support:
Handle tickets (incidents/requests) for data pipelines/chatbot applications & work with product owners/business customers to track the tickets to closure within pre-defined SLAs.
