Data Engineer

Role & responsibilities

• 6+ years of experience in software development, data engineering, ETL, and analytics reporting development
• Experience in building and maintaining data and system integrations using dimensional data modelling and optimized ETL pipelines.
• Proven track record of designing and implementing complex data solutions
• Demonstrated understanding and experience using:
• Python & PySpark: Ability to write efficient scripts for data transformation, and pipeline orchestration, knowledge of Spark or any distributing frameworks
• Knowledge of data modelling and data architecture principles
• Familiarity with event-driven architectures , API -based data sources , data quality validation, archival strategies, and incremental loading.
• Distributed Data Framework (e.g., Spark). Good understanding of Spark architecture
• Cloud platform services (AWS / Azure). AWS services (AWS Glue, Lambda, S3 , Stepfunctions)
• Proficiency in SQL and relational database management systems
• Optimize and tune database performance, including indexing, query optimization, and storage management.
•    CICD - Terraform or CloudFormation, Jenkins ,Bitbucket : For infrastructure-as-code and deployment automation.
• Experience with database design concepts: schema definition, normalization, relationships, indexing, constraints.
• Knowledge of Data lakes, Data warehouses
• Databricks/Delta Lakehouse architecture