Pyspark Developer

.

• Strong expertise in Python programming:

  Deep knowledge of Python language features, libraries, and best practices.

• Extensive experience with Apache Spark and PySpark:

  Writing optimized Spark jobs, transformations, actions, and working with RDDs and DataFrames.

• Solid understanding of big data processing concepts:

  Distributed computing, fault tolerance, partitioning, and data shuffling.

• Experience with ETL pipeline development:

  Designing, implementing, and maintaining scalable data pipelines.

• Proficient in working with large datasets:

  Handling data ingestion, cleansing, and transformation efficiently.

• Good understanding of SQL:

  Writing complex SQL queries and integrating Spark SQL where needed.

• Hands-on experience with FastAPI:

  Developing high-performance, scalable RESTful APIs and microservices using FastAPI.

• Experience with cloud platforms and services:

  Working knowledge of AWS (EMR, S3), Azure, or GCP cloud environments.

• Familiarity with Hadoop ecosystem components:

  HDFS, Hive, HBase, or similar tools.

• Knowledge of data serialization formats:

  JSON, Parquet, Avro, ORC.

• Strong debugging and troubleshooting skills:

  Ability to profile, optimize, and debug Spark and API applications.

• Experience with version control systems:

  Git or similar tools.

• Understanding of software development lifecycle and Agile methodologies:

  Participating in code reviews, CI/CD pipelines, and sprint planning.

• Excellent problem-solving and communication skills:

  Ability to collaborate with data engineers, analysts, and stakeholders.