Data Engineer

We are seeking a skilled Data Engineer with extensive experience in the Cloudera Data Platform (CDP) to join our dynamic team. The ideal candidate will have over four years of experience in designing, developing, and managing data pipelines, and will be proficient in big data technologies. This role requires a deep understanding of data engineering best practices and a passion for optimizing data flow and collection across a diverse range of sources. 

Required Skills and Qualifications:

• Experience:

  4+ years of experience in data engineering, with a strong focus on big data technologies. 

• Cloudera Expertise:

  Proficient in Cloudera Data Platform (CDP) and its ecosystem, including Hadoop, Spark, HDFS, Hive, Impala, and other relevant tools. 

• Programming Languages:

  Strong programming skills in Python, Scala, or Java. 

• ETL Tools:

  Experience with ETL tools and processes. 

• Data Warehousing:

  Knowledge of data warehousing concepts and experience with data modeling. 

• SQL:

  Advanced SQL skills for querying and manipulating large datasets. 

• Linux/Unix:

  Proficiency in Linux/Unix shell scripting. 

• Version Control:

  Familiarity with version control systems like Git. 

• Problem-Solving:

  Strong analytical and problem-solving skills. 

• Communication:

  Excellent verbal and written communication skills, with the ability to explain complex technical concepts to non-technical stakeholders. 

Preferred Qualifications:

• Cloud Experience:

  Experience with cloud platforms such as AWS, Azure, or Google Cloud. 

• Data Streaming:

  Experience with real-time data streaming technologies like Kafka. 

• DevOps:

  Familiarity with DevOps practices and tools such as Docker, Kubernetes, and CI/CD pipelines. 

Education:

• Bachelor’s degree in computer science, Information Technology, or a related field. 

Domain Skill:

Data Engineer 

Main Skill:

Hadoop, Spark,Hive,Impala,Scala,Python,Java,Linux 

1. Design and Development:

    

• Develop and maintain scalable data pipelines using Cloudera Data Platform (CDP) components. 
• Design and implement ETL processes to extract, transform, and load data from various data sources into the data lake or data warehouse. 
• Optimize and troubleshoot data workflows for performance and efficiency. 

1. Data Management:

    

• Manage and administer Hadoop clusters within the Cloudera environment. 
• Monitor and ensure the health and performance of the Cloudera platform. 
• Implement data security best practices, including encryption, data masking, and user access control.