Job
Description
Mandatory Skill Sets for Lead Data Engineer: 1. ETL Data Pipeline Development Strong experience with ETL tools : Informatica, AWS Glue, Apache Airflow, Talend, Azure ADF Hands-on experience in data ingestion, wrangling, transformation, and orchestration Ability to build real-time and batch data pipelines 2. Cloud Data Engineering (AWS/Azure/GCP) Proficiency in AWS (S3, Glue, Lambda, Kinesis, Redshift, EMR, IAM) Experience with Azure Data Factory (ADF), Synapse, Event Hub, Blob Storage Familiarity with GCP DataProc, BigQuery, Dataflow 3. Big Data Streaming Technologies Expertise in Apache NiFi, Kafka, Spark for streaming and real-time processing Hands-on experience in Big Data processing frameworks (Hadoop, Databricks, Delta Lake) 4. Programming Scripting Strong coding skills in Python, Java, or Scala Proficiency in SQL for complex queries, analytics, and performance tuning 5. Database Storage Hands-on experience with SQL and NoSQL databases (PostgreSQL, MongoDB, Cassandra, DynamoDB) Strong understanding of data modeling and optimization techniques 6. DevOps, CI/CD Orchestration Experience with CI/CD pipelines for data deployment Knowledge of Docker, Kubernetes for containerized data processing 7. Data Security Governance Strong understanding of data security, compliance, and governance frameworks Hands-on experience implementing access controls, encryption, and data privacy measures 8. Leadership Team Management Prior experience leading and mentoring data engineering teams Ability to collaborate with architects, developers, and business stakeholders Preferred/Good-to-Have Skills: Experience with Snowflake, BigQuery, Redshift, Synapse Hands-on experience with cost optimization of cloud data platforms Knowledge of Machine Learning pipelines and MLOps
