Data Engineer

As a Data Engineer, you will be responsible for developing, maintaining, and optimizing robust data ingestion and ETL/ELT pipelines using Python on the Databricks platform. You will work collaboratively with the architecture team to align implementation with long-term data strategies. Leveraging Delta Lake for data storage optimization and performance tuning will be a key part of your role. Additionally, you will utilize Prometheus for monitoring the health and performance of data pipelines and services. Ensuring quality, reliability, and scalability of the data platform across multiple domains and use cases is crucial. You will also drive automation and reusability in data processing pipelines to reduce manual overhead. Participating in code reviews, peer programming, and continuous improvement efforts will be expected from you. Adherence to best practices in coding, security, and documentation is essential. Engaging in team standups and collaborative sessions in the office for 3 days a week is part of the routine. Qualifications Required: - 5+ years of professional experience in Python development, primarily in data engineering use cases. - Strong experience working with Databricks and Delta Lake. - Proficiency in SQL, data manipulation, and working with large-scale structured and semi-structured datasets. - Hands-on experience with monitoring tools like Prometheus. - Solid understanding of modern data architecture, especially around data lakes and distributed processing. - Deep understanding of scalable system design, performance optimization, and best coding practices. - Familiarity with CI/CD workflows and Git-based version control. Additional Details: - Experience with Spark (especially PySpark). - Exposure to other monitoring tools like Grafana, ELK stack, etc. - Knowledge of cloud platforms such as AWS, Azure, or GCP. - Understanding of data governance and security practices in enterprise environments. Skills: - Strong analytical and problem-solving skills. - Excellent verbal and written communication. - Ability to collaborate effectively across globally distributed teams. - Proactive, self-motivated, and innovative mindset.,