Big Data Developer

About the Role:

Big Data Developer

Key Responsibilities:

1. Big Data Development

• Design, develop, and maintain

  big data processing solutions

  using frameworks such as Hadoop, Spark, Hive, or Kafka.
• Implement ETL pipelines for ingestion, transformation, and storage of large datasets.
  
• Optimize data pipelines for performance, scalability, and reliability.
  

2. Data Integration & Management

• Integrate data from multiple sources including relational databases, APIs, and streaming platforms.
  
• Ensure

  data quality, consistency, and security

  across all systems.

3. System Architecture & Optimization

• Design and implement

  data storage solutions

  including data lakes and data warehouses.
• Optimize system performance for large-scale data processing and retrieval.
  
• Maintain documentation for data workflows, pipelines, and system architecture.
  

4. Innovation & Continuous Improvement,

• Evaluate and recommend new technologies and tools to improve data processing capabilities.
  
• Implement best practices for coding, testing, and deploying big data solutions.

Preferred Qualifications:

• Experience in

  real-time data processing

  and streaming solutions.
• Familiarity with

  data warehousing concepts

  and tools (Redshift, Snowflake, BigQuery).
• Knowledge of containerization and orchestration tools (Docker, Kubernetes).
  
• Exposure to machine learning pipelines and predictive analytics.
  
• Big Data certification (Cloudera, Hortonworks, AWS Big Data Specialty, etc.) is an advantage.