Data Engineer (Pyspark+SQL)

As a Data Solutions Designer and Implementer, your role involves designing and implementing high-performance, scalable data solutions for large enterprise-wide data mining and processing. You will be responsible for designing data flows, deploying Big Data Platforms, and proposing end-to-end data pipelines for data projects. Your expertise in Databricks, Spark, and SQL will be crucial for the success of data initiatives. Key Responsibilities: - Designing and proposing end-to-end data pipelines for data projects and databases supporting web-based applications - Implementing data warehouses and data marts to serve data consumers - Executing the implementation of designs, transitioning from design stages to operationalization and maintenance - Database design and modeling at both macro and micro levels, providing logical data models - Database performance tuning and data lifecycle management - Assisting in the support and enhancement of existing data pipelines and databases Qualifications Required: - 6-10 years of experience working with data integration teams - 3+ years of experience developing data pipelines in an Apache Spark environment (preferably Databricks) - 2+ years of active work with Databricks, demonstrating in-depth experience - 2+ years of experience in working with a data warehouse and knowledge of data warehouse modeling techniques - Strong knowledge of Pyspark, Python, SQL, and distributed computing principles - Strong knowledge of data modeling, database technologies, and data warehousing - Experience with ETL/ELT processes, design, and implementation via SSIS or other ELT/ETL tools - Knowledge of cloud platforms (AWS or Azure) and big data technologies (Hadoop, Spark, etc.) - Fluent in complex SQL query performance tuning and database performance tuning - Understanding the importance of performance and ability to implement best practices for data-centric projects Additional Company Details: The company values experience in developing data solutions using native IaaS and PaaS solutions on AWS (Redshift, RDS, S3) as an advantage. This job offers you the opportunity to work on cutting-edge data solutions, leveraging your expertise in data integration, Apache Spark, Databricks, and data warehouse modeling techniques. Your role will be crucial in ensuring high-performance, scalable data solutions for the organization's data initiatives.,