Databricks/Pyspark Developer

Job Summary:
Looking for an offshore Senior Developer who has experience in Databricks/PySpark, is willing to learn new technologies if needed and is able to work with a team.

Essential Job Functions:

• Design and development of data ingestion pipelines (Databricks background preferred).
• Performance tune and optimize the databricks jobs.
• Evaluated new features and refractors existing code.
• Develop and integrate software applications using suitable development methodologies and standards, applying standard architectural patterns, taking into account critical performance characteristics and security measures.
• Collaborate with Business Analysts, Technical Manager, Architects and Senior Developers to establish the physical application framework (e.g. libraries, modules, execution environments).
• Perform end to end automation of ETL process for various datasets that are being ingested into the big data platform.
• Must be willing to flex work hours accordingly to support application launches and manage production outages if necessary.
• Works on best practices and documenting the process code merges and releases (Bitbucket).
• Works with architects and managers on designs and best practices. Responsible for design of application considering the cost and best practices.

Good data analysis skills.

• Must be willing to self learn new technologies, become SMEs and develop high quality code in a fast paced environment.
• Mentor junior developers and be hands on in development work.
• Work with QA and automation team. Must have attention to detail to cover all scenarios for testing.
• Handles and supports current production applications.
• Performs code reviews and discussion on the changes with Technology Manager.

Minimum Qualifications and Job Requirements:

• Must be a team player.
• Must have at least 5 years of IT development experience.
• Must have strong analytical and problem-solving skills.
• Must have experience in designing solutions, performing code reviews, mentoring junior engineers.
• Must have strong SQL and backend experience, and work on data driven projects.
• Experience collaborating with dev team, project managers, and engineers.
• Excellent communication and teamwork skills

Must have following experience:

• Python/PySpark
• SQL
• Databricks
• SCALA
• SQL
• Spark/Spark Streaming
• Big Data Tool Set
• Linux
• Kafka