Data Engineer

Position Details:

The Data Engineer is responsible for designing, developing, and maintaining robust, scalable, and secure data pipelines and platforms that enable efficient storage, processing, and consumption of data across the enterprise. The role ensures high-quality, timely, and governed data delivery to data scientists, analysts, and business users.

The Data Engineer plays a crucial role in building cloud-native data architectures, integrating data from diverse sources, and supporting the full lifecycle of data—from ingestion to transformation to consumption. The role ensures that the data infrastructure is reliable, compliant, and optimized to support advanced analytics and AI use cases across business functions.

• Qualification:

  Technical graduate (Engineering
  degree). Certifications in cloud data platforms (Azure Data Engineer, AWS Big Data, GCP Data Engineer), Big Data tools, or database technologies are preferred

• Experience:

  5–10 years of experience in data engineering or related roles. Proven experience in building and optimizing data pipelines, cloud data architecture, and handling large-scale datasets. Experience working with cross-functional data science, analytics, and IT teams.

Key Responsibilities

Data Pipeline Development

• Work Design, develop, and manage scalable ETL/ELT pipelines to extract, transform, and load data
• Automate ingestion from multiple sources including APIs, files, databases, and third-party systems
• Implement data ingestion from a variety of sources: ERP, CRM, APIs, file systems, IoT, etc.
• Implement efficient batch and real-time data processing pipelines

Data Integration & Middleware Interaction

• Integrate data from source systems via middleware platforms and API-based connectors
• Leverage OData, REST, SOAP, and other data protocols to extract data from third-party platforms (e.g., SAP, Salesforce, legacy systems)
• Collaborate with integration teams to establish secure and reliable connections via middleware (e.g., MuleSoft, Azure Integration Services)

Data Modelling & Architecture

• Design and implement data models for operational and analytical use cases (e.g., star/snowflake schemas) . Leverage knowledge of source systems schemas for ETL design and development
• Work closely with Data Architects and Analysts to ensure alignment with business needs
• Optimize data storage using partitioning, compression, and indexing strategies

Cloud Data Platform Engineering

• Build and manage cloud-native data solutions using services like Azure Data Factory, AWS Glue, or GCP Dataflow
• Ensure optimal performance, scalability, and cost-efficiency of cloud data environments
• Support migration from legacy systems to modern cloud-based architectures

Data Quality & Governance

• Implement data validation, reconciliation, and monitoring frameworks
• Ensure data accuracy, completeness, and integrity through automated checks
• Collaborate with data governance teams to ensure metadata management and lineage tracking
• Data Lineage- Document lineage and transformations to support transparency and auditability

Collaboration and Support

• Partner with Data Scientists, Analysts, and BI Developers to deliver data for ML models, dashboards, and reports
• Provide support for data issues and troubleshoot pipeline failures
• Document code, workflows, and platform configurations for maintainability

Security & Compliance

• Ensure secure access control, encryption, and compliance with data privacy regulations
• Work with security teams to perform vulnerability assessments on data systems
• Implement logging and monitoring for audit and traceability

Technology Background

• Infrastructure & Systems:

  Build and Maintain the data pipelines, databases, and other systems that an organization uses to manage its data

• Data Processing & Transformation:

  Develop processes to transform raw data into a usable format for analysis and reporting

• Programming Languages:

  Proficiency in languages Python, SQL, Scala, Shell scripting

• ETL Tools:

  Experience on ETL tools - Azure Data Factory, Apache NiFi, Talend, Informatica

• Big Data Frameworks:

  Awareness of Apache Spark, Hadoop, Kafka

• Cloud Platforms:

  Awareness of Azure (preferred), AWS, GCP

• Data Storage:

  Azure Data Lake, Delta Lake, Snowflake, AWS S3, BigQuery

• Databases:

  MS SQL Server, PostgreSQL, MySQL, MongoDB

• Orchestration:

  Airflow, Azure Data Factory, Prefect

• CI/CD & Version Control:

  Git, Azure DevOps, Jenkins

• Monitoring & Logging:

  Datadog, Prometheus, ELK Stack

• Visualization:

  Awareness of few visualization tools example- power BI etc

• Data Governance Tools:

  Responsible for ensuring that all data is stored securely and that investments in security measures are made and regularly maintained example Collibra, Microsoft Purview, Alation (as applicable)

About Company:

Coromandel International Limited

Founded in the early 1960s (as Coromandel Fertilisers), the company is currently headquartered in Chennai with its registered office in Hyderabad.

They are one of India’s largest private-sector producers of phosphatic fertilizers and the world’s largest manufacturer of neem-based bio-pesticides. Additionally, they lead the market in organic fertilizers and operate the country’s largest agri-retail chain, with 1000+ stores serving over 2 crore farmers.