Posted:1 day ago|
Platform:
Work from Office
Full Time
A Senior Engineer - Software / ETL Developer with PySpark is responsible for designing, developing, and optimizing data pipelines and ETL processes using PySpark and other related big data technologies. This role typically involves working with large-scale datasets and ensuring data quality, integrity, and performance.
Design, develop, and maintain robust and scalable ETL pipelines using Python and PySpark to extract, transform, and load data from various sources into data warehouses or data lakes.
Implement complex data transformations, aggregations, and optimizations within Spark for efficient data processing and analysis.
Work with various components of the big data ecosystem, potentially including Apache Spark, Hadoop, Kafka, and cloud-based data platforms (e.g., AWS EMR, Google Cloud Dataflow, Azure Databricks).
Optimize Spark jobs and ETL processes for performance, efficiency, and resource utilization.
Implement data quality checks, validation rules, and ensure data consistency and accuracy across pipelines.
Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and translate them into technical solutions.
Develop and maintain automated data workflows using tools like Apache Airflow or cloud-native orchestration services.
Troubleshoot and resolve issues related to data pipelines, performance bottlenecks, and data quality.
Provide technical guidance and mentorship to junior team members, and potentially lead small projects or initiatives.
SLK Digital
Upload Resume
Drag or click to upload
Your data is secure with us, protected by advanced encryption.
Browse through a variety of job opportunities tailored to your skills and preferences. Filter by location, experience, salary, and more to find your perfect fit.
We have sent an OTP to your contact. Please enter it below to verify.
Instantly access job listings, apply easily, and track applications.
Practice Python coding challenges to boost your skills
Start Practicing Python Now
bengaluru
5.0 - 9.0 Lacs P.A.
bengaluru
5.0 - 9.0 Lacs P.A.
bengaluru
5.0 - 9.0 Lacs P.A.
5.0 - 9.0 Lacs P.A.
bengaluru
5.0 - 9.0 Lacs P.A.
bengaluru
5.0 - 9.0 Lacs P.A.
bengaluru
5.0 - 9.0 Lacs P.A.
bengaluru
5.0 - 9.0 Lacs P.A.
5.0 - 9.0 Lacs P.A.
bengaluru
5.0 - 9.0 Lacs P.A.
