Big Data Engineer

Job Overview:

• Data Architecture & Design:

  Design and implement scalable data architectures, pipelines, and workflows to handle large volumes of structured and unstructured data.

• Big Data Processing:

  Develop and maintain data processing frameworks using PySpark to handle real-time and batch processing needs.

• Database Management:

  Write efficient SQL queries for data extraction, transformation, and analysis. Manage and optimize Hive tables and queries.

• Cloud Integration:

  Utilize AWS or Azure services (such as S3, Redshift, Azure Data Lake, Azure SQL) for data storage, processing, and analytics. Ensure data security and compliance with best practices.

• Data Quality & Optimization:

  Implement data validation and quality checks to ensure accuracy and consistency. Optimize data processing workflows for performance and cost-efficiency.

• Collaboration:

  Work closely with data scientists, analysts, and other engineering teams to understand data requirements and deliver solutions that meet business needs.

• Troubleshooting:

  Diagnose and resolve data processing issues, ensuring minimal downtime and disruption to data workflows.

• Documentation:

  Maintain clear and comprehensive documentation of data pipelines, processes, and systems.

Required Qualifications:

• Education:

  Bachelors degree in Computer Science, Engineering, Mathematics, or a related field. Masters degree is a plus.

• Experience:

  3+ years of experience as a Big Data Engineer or in a similar role with a proven track record of working on large-scale data projects.

• Technical Skills:

• Proficiency in Python Coding for data manipulation and processing.
• Strong experience with PySpark for distributed data processing.
• Advanced SQL Coding skills for querying and managing relational databases.
• Experience with Hive for data warehousing and querying.
• Hands-on experience with cloud platforms (AWS or Azure) and their data services.

Note : This is completely Work From Office and 1 PM to 10 PM