Senior Data Engineer

Overall Responsibilities

• Design, develop, and maintain scalable data pipelines and architectures to support analytical and operational needs
• Implement data integration solutions, ensuring high data quality, security, and compliance standards
• Collaborate with data scientists, analysts, and engineering teams to translate business requirements into technical solutions
• Optimize data workflows for performance, reliability, and cost-efficiency in cloud environments
• Develop and enforce best practices for data governance, security, and privacy
• Lead efforts in automating data processes and providing documentation for data systems
• Monitor data system performance, troubleshoot issues, and implement improvements
• Contribute to strategic planning around data infrastructure and technology adoption

Technical Skills

• Programming Languages:
  • Required: Python, SQL (e.g., T-SQL, PL/SQL)
  • Preferred: Scala, Java
• Databases/Data Management:
  • Required: Relational databases, data warehousing solutions
  • Preferred: NoSQL databases (e.g., MongoDB, Cassandra)
• Cloud Technologies:
  • Required: Hands-on experience with AWS, Azure, or Google Cloud Platform services for data processing and storage
  • Preferred: Cloud architecture design, multi-cloud strategies
• Frameworks and Libraries:
  • Required: Spark (PySpark, Spark SQL)
  • Preferred: Kafka, Hadoop ecosystem components
• Development Tools and Methodologies:
  • Required: Git, CI/CD pipelines, Agile/Scrum practices
  • Preferred: Jenkins, Terraform, Docker, Kubernetes
• Security Protocols:
  • Required: Data encryption, access controls, compliance standards (e.g., GDPR, HIPAA)

Experience Requirements

• A minimum of 6 years in data engineering or related roles
• At least 4 years of experience designing and building large-scale data pipelines and systems
• Proven experience working within cloud-based data architectures
• Hands-on project leadership in data integrations, migrations, or large data initiatives
• Experience in dynamic, fast-paced environments with evolving data needs
• Knowledge of industry best practices in data security and compliance

Day-to-Day Activities

• Develop, test, and deploy data pipelines managing structured and unstructured data sources
• Collaborate in daily stand-ups, planning sessions, and cross-team meetings to align project goals
• Monitor system performance, troubleshoot issues, and implement improvement strategies
• Automate routine data processes while ensuring data quality and security standards
• Document data workflows, architecture designs, and operational procedures
• Engage with stakeholders to gather requirements and translate them into technical solutions
• Review and mentor junior team members, fostering a collaborative knowledge-sharing environment

Qualifications

• Bachelors or Masters degree in Computer Science, Data Science, Information Systems, or a related field
• Professional certifications such as AWS Certified Data Analytics, Google Cloud Professional Data Engineer, or similar are preferred
• Demonstrated commitment to continuous professional development through training and industry engagement

Soft Skills

• Strong analytical and problem-solving capabilities with a focus on data integrity and quality
• Effective leadership skills and proven ability to work collaboratively in multidisciplinary teams
• Excellent verbal and written communication skills for technical and non-technical audiences
• Ability to adapt to changing priorities, manage multiple projects, and meet deadlines
• Demonstrates innovative thinking and openness to emerging data technologies
• Prioritizes tasks efficiently, balancing strategic initiatives with operational demands.