Data Engineer

4
Job Role & Responsibilities Role:
Data Engineering & Quality Management : Design, implementation, and management of data platform, data pipelines to help fulfill data access and aggregation functionality for users in cloud environment. Develop and Maintain Data Pipelines: Design, build, and manage scalable data pipelines for data extraction, transformation, and loading (ETL).
Data Integration: Integrate data from various sources, ensuring data quality and consistency.
Optimize Data Systems: Identify and implement process improvements to enhance data reliability, efficiency, and quality.
Collaborate with Teams: Work with data scientists, analysts, and other stakeholders to support their data infrastructure needs.
Monitor and Troubleshoot: Monitor data pipelines and systems, troubleshoot issues, and ensure data integrity.
Documentation: Maintain comprehensive documentation of data processes and systems.
5 Competency Requirements Technical/ Functional:
Experience: Minimum 5 years of experience in data engineering or a related field.
Minimum 3 years of experience in Databricks platform. Technical Skills: Proficiency in Py-Spark (Must Have), SQL, and Python.
Storage Management: Experience in storage mechanism in AWS S3, Layers of Medallion Architecture, OTF like Delta, iceberg etc., Hand-Ons in Data schema like Star, snowflake etc.
§ Experience in IoT time series data domain. § Ability to work independently or within a team, proactively, in a fast-paced AGILE-SCRUM environment. Desired but not essential : § Exposure to automotive data Behavioral § Effective team player. § Study industry trends (global and domestic) and apply it in work-area. § Co-ordinate department activities and represent department / MSIL in forums. § Good communication and presentation skills.