Senior Data Engineer

Senior Data Engineer We are seeking a highly skilled Senior Data Engineer with strong expertise in big data, Spark, and modern cloud-native platforms. The ideal candidate will work closely with data architects, platform teams, and vendor products to build scalable, sustainable, and performant data pipelines and APIs in a dynamic offshore environment. Location: Offshore (Gurugram) Job specific Duties and Responsibilities Design, develop, and optimize data pipelines for processing large volumes of structured and unstructured data using Apache Spark and AWS technologies. Develop APIs and microservices using container frameworks like OpenShift, Docker, and Kubernetes. Work with diverse data formats such as PARQUET, ORC, and CSV. Leverage data streaming and messaging platforms including Apache Kafka for real-time data processing. Build scalable solutions on AWS leveraging services like ElasticSearch/OpenSearch. Implement big data querying using tools such as Presto or Trino. Collaborate with platform and vendor deployment teams to ensure seamless integration of data sources. Work closely with data architects to provision and support sustainable infrastructure patterns. Contribute to data access strategies and data modeling in alignment with architectural principles. Communicate technical concepts effectively to non-technical stakeholders and vice versa. Required Competencies and Skills Advanced data engineering skills with strong experience using Spark, PySpark, SQL (Oracle/PostgreSQL or MySQL), Python, Kafka, and Airflow Experience in building and delivering API based microservices solutions using container frameworks like OpenShift, Docker, or Kubernetes Experience in various file format types like PARQUET, ORC, CSV Experience with AWS and technologies such as Elastic or Opensearch Strong knowledge of big data querying tools such as Presto or Trino Experience in data architecture principles, including data access patterns and data modelling Previously worked closely with data architect and platform/infrastructure teams to develop and provision sustainable infra patterns Design, develop and optimise data pipelines for processing large volumes of structured and unstructured data using Apache Spark and AWS technologies Work closely with platform and vendor product deployment teams to ensure seamless integration of data sources Excellent at stakeholder management across multiple levels of engagement Proactive and have great communication skills Good in doing downstream analysis and understanding end to end data flow when multiple systems are involved A natural collaborator with a learning mindset, happy to share knowledge and learn from others Able to understand technical requirements and translate them into non-technical language and vice versa Experienced with working on Agile delivery Familiarity with data governance, data quality and control frameworks would certainly be useful in this role Able to creatively use data and insights to uncover new opportunities, identify root causes and underlying risks to recommend solutions to the business Required Experience and Qualifications 6+ years of professional experience in Data Engineering and Microservices development. Proven experience with SPARK, SCALA, AWS, and modern data platforms. Strong experience working in Agile delivery environments. Bachelors or Masters degree in Computer Science, Engineering, or a related quantitative field. Prior experience in Financial Services or Banking is a plus. AWS or equivalent cloud certification preferred..