Job
Description
We are looking for a results-driven Senior Data Engineer to join our engineering team. The ideal candidate will have hands-on expertise in data pipeline development, cloud infrastructure, and BI support, with a strong command over modern data stacks. You'll be responsible for building scalable ETL/ELT workflows, managing data lakes and marts, and enabling seamless data delivery to analytics and business intelligence teams. This role requires deep technical know-how in PostgreSQL, Python scripting, Apache Airflow, AWS or other cloud environments, and a working knowledge of modern data and BI tools Responsibilities Design and optimize complex SQL queries, stored procedures, and indexes. Perform performance tuning and query plan analysis. Contribute to schema design and data normalization. Migrate data from multiple sources to cloud or ODS platforms. Design schema mapping and implement transformation logic. Ensure consistency, integrity, and accuracy in migrated data. Build automation scripts for data ingestion, cleansing, and transformation. Handle file formats (JSON, CSV, XML), REST APIs, cloud SDKs (e. g., Boto3). Maintain reusable script modules for operational pipelines. Develop and manage DAGs for batch/stream workflows. Implement retries, task dependencies, notifications, and failure handling. Integrate Airflow with cloud services, data lakes, and data warehouses. Manage data storage (S3 GCS, Blob), compute services, and data pipelines. Set up permissions, IAM roles, encryption, and logging for security. Monitor and optimize the cost and performance of cloud-based data operations. Design and manage data marts using dimensional models. Build star/snowflake schemas to support BI and self-serve analytics. Enable incremental load strategies and partitioning. Work with tools like DBT, Fivetran, Redshift, Snowflake, BigQuery, or Kafka. Support modular pipeline design and metadata-driven frameworks. Ensure high availability and scalability of the stack. Collaborate with BI teams to design datasets and optimize queries. Support the development of dashboards and reporting layers. Manage access, data refreshes, and performance for BI tools. Requirements 4-6 years of hands-on experience in data engineering roles. Strong SQL skills in PostgreSQL (tuning, complex joins, procedures). Advanced Python scripting skills for automation and ETL. Proven experience with Apache Airflow (custom DAGs, error handling). Solid understanding of cloud architecture (especially AWS). Experience with data marts and dimensional data modeling. Exposure to modern data stack tools (DBT, Kafka, Snowflake, etc. ) Familiarity with BI tools like Power BI, Apache Superset, or Supertech BI. Version control (Git) and CI/CD pipeline knowledge are a plus. Excellent problem-solving and communication skills. This job was posted by Suryansh Singh Karchuli from ShepHertz Technologies. Show more Show less
