Data Engineer

Job Description

Description The Data Engineer will help build and maintain the cloud Data Lake platform leveraging Databricks. Candidates will be expected to contribute to all stages of the data lifecycle including data ingestion, data modeling, data profiling, data quality, data transformation, data movement, and data curation Architect data systems that are resilient to disruptions and failures Ensure high uptime for all data services Bring modern technologies and practices into the system to improve reliability and support rapid scaling of the business’s data needs Scale up our data infrastructure to meet business needs Develop production data pipeline patterns leveraging Provide subject matter expertise and hands-on delivery of data acquisition, curation and consumption pipelines on Azure. Responsible for maintaining current and emerging state-of-the-art computer and cloud-based solutions and technologies. Build effective relationships with internal stakeholders Familiarity with the technology stack available in the industry for metadata management: Data Governance, Data Quality, MDM, Lineage, Data Catalog, etc. Hands-on experience implementing analytics solutions leveraging Python, Spark SQL, Databricks Lakehouse Architecture, Kubernetes, Docker All other duties as assigned Qualifications Bachelor's degree in Computer Science, Information Technology, Management Information Systems (MIS), Data Science or related field. Applicable years of experience may be substituted for the degree requirement. Up to 8 years of experience in software engineering Experience with large and complex data projects, preferred Experience with large-scale data warehousing architecture and data modeling, preferred Worked with Cloud-based architecture such as Azure Cloud, preferred Experience working with big data technologies e.g. Snowflake, Redshift, Synapse, Postgres, Airflow, Kafka, Spark, DBT, preferred Experience implementing pub/sub and streaming use cases, preferred Experience in design reviews, preferred Experience influencing a team’s technical and business strategy by making insightful contributions to team priorities and approaches, preferred Working knowledge of relational databases, preferred Expert in SQL and high-level languages such as Python, Java or Scala, preferred Demonstrate the ability to analyze large data sets to identify gaps and inconsistencies in ETL pipeline and provide solutions for pipeline reliability and data quality, preferred Experience in infrastructure as code / CICD development environment, preferred Proven ability to build, manage and foster a team-oriented environment Excellent communication (written and oral) and interpersonal skills Excellent organizational, multi-tasking, and time-management skills Job Engineering Primary Location India-Maharashtra-Mumbai Schedule: Full-time Travel: No Req ID: 244483 Job Hire Type Experienced Not Applicable #BMI N/A Show more Show less