Senior Data Engineer

Job Title: Senior Data Engineer

Location: Hyderabad(Client Office)

Job Type:

Years of Experience 5-8 years

About Straive:

Straive is a market leading Content and Data Technology company providing data services, subject matter expertise, & technology solutions to multiple domains. Data Analytics & Al Solutions, Data Al Powered Operations and Education & Learning form the core pillars of the companys long-term vision. The company is a specialized solutions provider to business information providers in finance, insurance, legal, real estate, life sciences and logistics. Straive continues to be the leading content services provider to research and education publishers. Data Analytics & Al Services: Our Data Solutions business has become critical to our client's success. We use technology and Al with human experts-in loop to create data assets that our clients use to power their data products and their end customers' workflows. As our clients expect us to become their future fit Analytics and Al partner, they look to us for help in building data analytics and Al enterprise capabilities for them. With a client-base scoping 30 countries worldwide, Straives multi-geographical resource pool is strategically located in eight countries - India, Philippines, USA, Nicaragua, Vietnam, United Kingdom, and the company headquarters in Singapore.

Website: https://www.straive.com/

Job Summary:

implement ETL/ELT solutions using Spark (3.x), Scala, SQL and Python

Key Responsibilities:

• Design, build and maintain scalable

  batch and (optionally) streaming data pipelines using Apache Spark 3.x, Scala, Python & SQL

• Implement core ETL/ELT logic in Scala and Python; author efficient Spark DataFrame/Dataset jobs.
• Write and optimize complex SQL for ingestion, transformation and consumption layers.

• Tune Spark jobs for performance and cost

  : partitioning, join strategies, broadcast, memory/tuning, shuffle reduction.
• Ensure code quality via unit tests, integration tests, CI/CD and code reviews.
• Work with data modeling, schema evolution and data quality checks to ensure reliable outputs.
• Collaborate with platform/DevOps teams to deploy and monitor pipelines (retries, logging, alerting).
• Troubleshoot production issues and perform root-cause analysis.
• Mentor and guide junior engineers, share best practices and drive improvements to the data platform.

Qualifications:

• 5–8 years industry experience as a Data Engineer.
• Strong hands-on experience with

  Apache Spark 3.x (development, tuning, debugging).

• Proficient in

  Scala programming for Spark ETL/ELT development

  (datasets, RDDs when required).
• Experience with streaming frameworks like

  Spark Structured Streaming, Kafka

  .
• Proficient in

  Python programming

  .

• Advanced SQL skills

  (complex queries, window functions, CTEs, query optimization).
• Experience building production-grade ETL/ELT workflows, testing and monitoring them.
• Experience with orchestration tools like Apache Airflow
• Strong software engineering fundamentals: version control (Git), modular code, code reviews, CI/CD pipelines.
• Experience with cloud object storage and common data formats (Parquet/ORC/Avro).
• Strong problem solving, communication and collaboration skills.
• Experience with AWS Glue, EMR, Athena, Redshift, S3 will be a plus.

Hybrid Work: Tue, Wed, Thu (3 days) office / 10.30 to 7. 30 IST work hours