Job
Description
Job Title: Data Engineer (3-5 Years Experience) Location: HYBRID- Mostly Remote Employment Type: Full-Time/ Contract Job Summary: We are seeking a highly skilled Data Engineer with 3-5 years of experience in designing, building, and optimizing data pipelines and cloud-based data solutions . The ideal candidate should have in-depth expertise in one of the following primary technologies: AWS, Azure, Databricks, or Snowflake while being familiar with other cloud and big data platforms. This role requires strong ETL development skills, data modeling knowledge, and a deep understanding of cloud-based data engineering best practices. Key Responsibilities: Design, develop, and optimize data ingestion and transformation pipelines using ETL/ELT frameworks . Implement big data processing solutions using Databricks (PySpark, Scala), Snowflake, AWS, or Azure . Develop and maintain structured and semi-structured data models , ensuring efficient data warehousing and analytics . Optimize data pipeline performance , handling large-scale datasets with efficiency. Ensure data governance, security, and compliance in cloud-based environments. Work with cloud-based storage, compute, and database services (e.g., AWS Redshift, Azure Synapse, Snowflake, Databricks Delta Lake). Collaborate with analysts, data scientists, and business teams to support reporting, analytics, and machine learning initiatives . Troubleshoot and resolve data pipeline and infrastructure issues . Implement monitoring and alerting mechanisms to maintain data pipeline health. Required Skills & Qualifications: 3-5 years of experience in data engineering, cloud computing, and ETL development . Primary expertise in one of the following: Databricks: Strong experience in Apache Spark, PySpark, Scala , and performance optimization. Snowflake: Deep knowledge of Snowflake architecture, SQL optimization, data sharing, and security best practices . AWS: Experience with AWS Glue, Redshift, S3, Lambda, and Step Functions for data workflows . Azure: Strong expertise in Azure Synapse, Data Factory, ADLS, and Databricks on Azure . Proficiency in SQL, Python, and data modeling techniques. Hands-on experience with structured and semi-structured data formats (JSON, Avro, Parquet, XML). Experience with data orchestration tools (Airflow, DBT, Prefect). Strong understanding of data security, governance, and compliance best practices. Excellent problem-solving and troubleshooting skills in big data environments . Preferred Qualifications: Experience with CI/CD pipelines for data engineering . Familiarity with real-time data streaming using Kafka, Kinesis, or Spark Streaming. Exposure to machine learning workflows and MLOps best practices. Knowledge of BI and visualization tools like Power BI, Looker, or Tableau.
