PySpark Developer

Job Summary:

Client is looking for PySpark Developer and this is Hybrid position in Hyderabad.

Responsibilities:

• Understands the full picture and helps with operations with monitoring, dashboards etc.
• Work with architect on design
• Constantly keeps an eye on cost and performance of the system
• Work with BA on the integration requirements
• Ability to trouble shoot performance issue and other functional issues
• Attention to details and documents the key decisions made
• Able to mentor or guide junior developers and do code reviews
• Ability to handle fast-paced projects and willing to look into production issues
• Document and maintain project artifacts.
• Maintain comprehensive knowledge of industry standards, methodologies, processes, and best practices.
• Promptly report any known or suspected loss, theft or unauthorized disclosure or use of PI to the General Counsel/Chief Compliance Officer or Chief Information Officer.
• Safeguard the company’s assets.
• Adhere to the company’s compliance program.
• Maintain comprehensive knowledge of industry standards, methodologies, processes, and best practices.
• Maintain focus on customer service, efficiency, quality, and growth.
• Collaborating with additional team members

Experience:

• Must have at least 5 years of IT development experience.
• Must have strong analytical and problem-solving skills.
• Designing, developing, and maintaining large-scale data processing applications using Apache Spark and Python to create scalable and efficient data pipelines for big data environments.
• Building ETL processes, optimizing data workflows, collaborating with data scientists and engineers, ensuring data quality, and implementing solutions for big data analytics and business intelligence.
• Must have strong SQL and backend experience and working on data driven projects.
• Must have following experience:
• Python/PySpark
• SQL
• Databricks
• SCALA
• SQL
• ETL
• Spark/Spark Streaming
• Big Data Tool Set
• Linux
• Kafka