Big Data Engineer

As a skilled and experienced Big Data Engineer, you will be responsible for building scalable data pipelines, managing data lakes, and optimizing large-scale data processing frameworks. Your role will involve: - Designing, developing, and maintaining scalable and efficient big data pipelines using PySpark and Spark. - Working with large datasets using Hadoop, Hive, and HDFS. - Developing data ingestion frameworks using Python and Spark to integrate structured and unstructured data from multiple sources. - Implementing data transformation, data validation, and data quality frameworks. - Optimizing performance of big data processing applications. - Working on cloud platforms (AWS) and managing data storage using S3. - Collaborating with cross-functional teams including data scientists, analysts, and business stakeholders. - Ensuring best practices in data security, governance, and compliance are followed. You must have: - Strong programming skills in Python and Scala. - Hands-on experience with Hadoop, Hive, and Spark. - Experience with PySpark and distributed computing. - Proficiency in working with AWS cloud services, especially S3 & EMR. - Strong knowledge of data structures, algorithms, and problem-solving. - Good understanding of data architecture and ETL pipelines. It would be nice to have: - Exposure to AWS services like EMR, Lambda, Glue. - Experience with workflow orchestration tools like Airflow or Oozie. - Familiarity with DevOps practices, CI/CD pipelines. - Knowledge of data warehousing concepts and tools like Redshift or Snowflake. Qualifications: - Bachelors/Masters degree in Computer Science, Engineering, or a related field. - 6+ years of relevant experience in Big Data technologies.,