Big Data Developer

Role & responsibilities

Preferred candidate profile

1. Experience Required:

   4 years

   Domain:

   Clinical & Healthcare( Main )

   Location:

   Bangalore

L

Job Responsibilities

• Design, develop, and optimize scalable big data pipelines using Hadoop, Spark, and related technologies.
• Build and maintain ETL workflows for ingesting, transforming, and processing large datasets from diverse sources.
• Implement data models and validation rules to ensure data quality and support advanced analytics.
• Collaborate with data analysts, business teams, and other stakeholders to deliver business intelligence solutions.
• Automate data validation, performance monitoring, and reporting using tools like Airflow, BigQuery, and Snowflake.
• Ensure data security and compliance (e.g., HIPAA, GDPR) in all data engineering activities.
• Troubleshoot and tune SQL queries, Spark jobs, and data pipelines for optimal performance.
• Document technical solutions, workflows, and best practices.

Required Skills

• Strong hands-on experience with Hadoop ecosystem (HDFS, Hive, Sqoop, Pig, Oozie).
• Proficiency in Spark (Scala/PySpark), Kafka, and ETL development.
• Solid knowledge of SQL, Python, and data warehousing concepts.
• Experience with cloud platforms (AWS, Azure, or GCP) and related services (e.g., S3, Redshift, Data Factory, Databricks).
• Familiarity with workflow orchestration tools (Airflow, Jenkins).
• Experience with data modeling, data validation, and performance tuning.
• Good understanding of data security, privacy, and compliance requirements.
• Excellent problem-solving and communication skills.

Preferred Qualifications

• Bachelors or Masters degree in Computer Science, Engineering, or related field.
• Experience in healthcare, banking, or other data-intensive domains is a plus.
• Certifications in Big Data technologies or cloud platforms are desirable