Senior Software Engineer II - Data Engineer- ETL, Pyspark

Primary Responsibilities:

• Lead all phases of data engineering, including requirements analysis, data modeling, pipeline design, development, and testing
• Design and implement performance and operational enhancements for scalable data systems
• Develop reusable data components, frameworks, and patterns to accelerate team productivity and innovation
• Conduct code reviews and provide feedback aligned with data engineering best practices and performance optimization
• Ensure data solutions meet standards for quality, scalability, security, and maintainability through rigorous design and code reviews
• Actively participate in Agile/Scrum ceremonies to deliver high-quality data solutions
• Collaborate with software engineers, data analysts, and business stakeholders across Agile teams
• Troubleshoot and resolve production issues post-deployment, designing robust solutions as needed
• Design, develop, test, and document data pipelines and ETL processes, enhancing existing components to meet evolving business needs
• Partner with architecture teams to drive forward-thinking data platform solutions
• Contribute to the design and architecture of secure, scalable, and maintainable data systems, clearly communicating design decisions to technical leadership
• Mentor junior engineers and collaborate on solution design with team members and product owners
• Comply with the terms and conditions of the employment contract, company policies and procedures, and any and all directives (such as, but not limited to, transfer and/or re-assignment to different work locations, change in teams and/or work shifts, policies in regards to flexibility of work benefits and/or work environment, alternative work arrangements, and other decisions that may arise due to the changing business environment). The Company may adopt, vary or rescind these policies and directives in its absolute discretion and without any limitation (implied or otherwise) on its ability to do so

Required Qualifications:

• Bachelors degree or equivalent experience
• Hands-on experience with cloud data services (AWS, Azure, or GCP)
• Experience building and maintaining ETL/ELT pipelines in enterprise environments
• Experience integrating with RESTful APIs
• Experience with Agile methodologies (Scrum, Kanban)
• Knowledge of data governance, security, privacy, and vulnerability management
• Understanding of authorization protocols (OAuth) and API integration
• Solid proficiency in SQL, NoSQL, and data modeling
• Proficiency with open-source tools such as Apache Flink, Iceberg, Spark, and PySpark
• Advanced Python skills for data engineering and data science (beyond Jupyter notebooks)
• Familiarity with big data technologies such as Spark, Hadoop, and Databricks
• Ability to build modular, testable, and reusable data solutions
• Solid grasp of data engineering concepts including:
  
• Data Catalogs
• Data Warehouses
• Data Lakes (especially Iceberg)
• Data Dictionaries

Preferred Qualifications:

• Experience with GitHub, Terraform, and GitHub Actions
• Experience with real-time data streaming (Kafka, Kinesis)
• Experience with feature engineering and machine learning pipelines (MLOps)
• Knowledge of data warehousing platforms (Snowflake, Redshift, BigQuery)
• Familiarity with AWS native data engineering tools:
  
• Lambda, Lake Formation, Kinesis (Firehose, Data Streams)
• Glue (Data Catalog, ETL, Streaming)
• SageMaker, Athena, Redshift (including Spectrum)
• Demonstrated ability to mentor and guide junior engineers