Back-End Engineer - SCALA Opening

About the Role:

We are seeking a skilled and passionate Back-End Engineer with a strong background in data engineering to join our growing team. In this role, you will be responsible for designing, developing, and maintaining scalable and reliable data pipelines and back-end systems. You will work with large datasets, leverage cloud technologies, and collaborate with data scientists and other engineers to deliver impactful data-driven solutions. This position is available in both the United States and India.

Responsibilities:

* Design, develop, and maintain data pipelines using Spark, Python, Scala, and Java.

* Write efficient and optimized SQL queries for data extraction, transformation, and loading (ETL) processes.

* Work with DataFrames to manipulate and analyze large datasets.

* Implement data storage and processing solutions using cloud technologies (preferably AWS or GCP).

* Build and maintain real-time data streaming pipelines using MSK/Kafka.

* Utilize S3 for data storage and retrieval.

* Work with data lake technologies like Iceberg.

* Ensure data quality, integrity, and security.

* Collaborate with data scientists and other engineers to understand data requirements and deliver solutions.

* Participate in code reviews and contribute to improving our development processes.

* Troubleshoot and resolve issues in data pipelines and back-end systems.

Required Skills:

Spark

* Expertise in SQL and working with relational databases.

* Experience with DataFrames for data manipulation and analysis.

* Experience with cloud technologies (preferably AWS or GCP).

* Experience with message streaming platforms like MSK/Kafka.

* Experience with S3 or similar object storage.

* Experience with data lake technologies like Iceberg.

* Solid understanding of data warehousing concepts.

* Excellent communication and collaboration skills.