Data Engineer

As a skilled Data Engineer with 3-5 years of experience, you will be responsible for developing, maintaining, and optimizing data infrastructure and pipelines. Your role will involve collaborating with data scientists, analysts, and stakeholders to ensure accessible, reliable, and efficiently processed data. Your key responsibilities will include designing, building, and managing scalable data pipelines for extraction, transformation, and loading (ETL). You will integrate data from various sources to maintain data quality and consistency. Identifying and implementing process improvements to enhance data reliability, efficiency, and quality will also be a crucial aspect of your role. Collaborating with data scientists, analysts, and other stakeholders to support their data infrastructure needs will be essential. You will be required to monitor data pipelines and systems, troubleshoot issues, and ensure data integrity. Additionally, maintaining comprehensive documentation of data processes and systems will be part of your responsibilities. To excel in this role, you should have 3-5 years of experience in data engineering or a related field. Proficiency in Py-Spark, SQL, Python, and data platforms like Databricks is a must. Experience in storage mechanisms in AWS S3, Layers of Medallion Architecture, OTF like Delta, iceberg, and hands-on experience in data schema like Star, snowflake are required. Strong analytical and problem-solving skills are essential, along with excellent communication and teamwork skills. Preferred qualifications include experience with cloud platforms (Databricks, AWS tools like Redshift, Glue, Airflow), familiarity with big data technologies (e.g., Hadoop, Spark), and experience working in Agile environments.,