Python & Databricks Engineer - Data Pipeline

Key Responsibilities

• Design, develop, and maintain scalable data pipelines using Databricks and Apache Spark.
• Write efficient Python code for data transformation, cleansing, and analytics.
• Collaborate with data scientists, analysts, and engineers to understand data needs and deliver high-performance solutions.
• Optimize and tune data pipelines for performance and cost efficiency.
• Implement data validation, quality checks, and monitoring.
• Work with cloud platforms (preferably Azure or AWS) to manage data workflows.
• Ensure best practices in code quality, version control, and documentation.
  

Required Skills & Experience

• 5+ years of professional experience in Python development.
• 3+ years of hands-on experience with Databricks (including notebooks, clusters, Delta Lake, and job orchestration).
• Strong experience with Spark (PySpark preferred).
• Proficient in working with large-scale data processing and ETL/ELT pipelines.
• Solid understanding of data warehousing concepts and SQL.
• Experience with Azure Data Factory, AWS Glue, or other data orchestration tools is a plus.
• Familiarity with version control tools like Git.
• Excellent problem-solving and communication skills.