Senior Data Engineer

Job Title

Location

Position Type

Job Summary

Key Responsibilities

• Design, develop, and maintain scalable data pipelines to collect, process, and store large volumes of data from various sources.
• Implement ETL (Extract, Transform, Load) processes to transform raw data into structured formats.
• Integrate data from multiple sources, including databases, APIs, third-party services, and data warehouses.
• Ensure data consistency and integrity across different data stores.
• Design and manage databases, data warehouses, and data lakes to support data storage and retrieval needs.
• Optimize database performance and ensure data security.
• Ability to write complex queries dealing with complex joins across numerous tables spanning multiple databases for efficient data pulls.
• Implement data validation and quality checks to ensure data accuracy and reliability.
• Monitor data pipelines and perform troubleshooting and debugging as needed.
• Work closely with data scientists, developers and business stakeholders to understand data requirements and deliver solutions.
• Communicate effectively with technical and non-technical team members.
• Document data pipelines, workflows, and processes.
• Establish and promote best practices for data engineering, including coding standards, version control, and testing.

Qualifications

• Bachelors degree in computer science, Engineering, Information Technology, or a related field. A master’s degree is a plus.
• 5+ years of experience in data engineering, software development, or a related field.
• Experience with data warehousing solutions such as Amazon Redshift, Google BigQuery, Snowflake, Azure Synapse or similar.
• Familiarity with big data technologies (e.g., Hadoop, Spark, Kafka).
• Proficiency in programming languages such as Python, Java, or Scala.
• Experience with SQL and relational databases (e.g., PostgreSQL, MySQL, SQL Server).
• Strong knowledge of ETL tools and frameworks (e.g., Apache NiFi, Apache Airflow, Talend, Informatica).
• Understanding of data modeling and database design principles.
• Excellent problem-solving and analytical skills.
• Strong communication and teamwork abilities.
• Ability to manage multiple tasks and prioritize effectively.