Databricks Data Engineer

As a Senior Data Engineer at our company, you will play a crucial role in designing, developing, and optimizing scalable data pipelines and applications using Apache Spark and Databricks. Your responsibilities will include: - Designing, developing, and implementing scalable data pipelines and applications with a focus on adhering to industry best practices. - Performing in-depth performance tuning and optimization of Spark applications within the Databricks environment. - Troubleshooting complex issues related to data ingestion, transformation, and pipeline execution. - Collaborating with cross-functional teams to deliver end-to-end data solutions. - Continuously evaluating and adopting new technologies and tools in the Databricks and cloud ecosystem. - Optimizing Databricks cluster configurations for cost-effectiveness and performance. - Applying data engineering principles to enable high-quality data ingestion, transformation, and delivery processes. - Documenting technical designs, development processes, and operational procedures. Qualifications required for this role are: - Bachelors degree in Computer Science, Engineering, or a related field. - 10+ years of experience in data engineering or big data development. - 5+ years of hands-on experience with Apache Spark and Databricks. - Deep understanding of Spark internals, Spark Streaming, and Delta Lake. - Experience in developing solutions using Azure Data Services including Azure Databricks, Azure Data Factory, Azure DevOps, Azure Functions, Azure SQL Database, Azure Event Grid, and Cosmos DB. - Proficiency in PySpark or Scala. - Strong experience in performance tuning, cost optimization, and cluster management in Databricks. - Solid understanding of data warehousing, ETL/ELT pipelines, and data modeling. - Experience working with cloud platforms (Azure preferred; AWS/GCP is a plus). - Familiarity with Agile/Scrum methodologies. Preferred qualifications include: - Databricks Certified Professional Data Engineer certification is a strong plus. - Strong communication skills, both written and verbal, with the ability to convey technical concepts to non-technical stakeholders.,