Data Engineer

Job Description

Experience Level: 5+ years Location: Bangalore Job Description: We are seeking an experienced Databricks + PySpark Developer with 5+ years of experience in building and maintaining large-scale data processing systems. You will be responsible for developing and optimizing data pipelines, leveraging Databricks and PySpark in a cloud-based environment. The ideal candidate should have a strong understanding of big data frameworks and experience working with cloud technologies. Key Responsibilities: Design, develop, and maintain ETL pipelines using PySpark and Databricks . Optimize data processing workflows to ensure scalability and efficiency. Integrate and process large datasets from various sources into cloud data lakes and data warehouses. Collaborate with data engineers, analysts, and data scientists to define data requirements and ensure the quality of data pipelines. Write clean, efficient, and reusable code for data transformations in PySpark . Troubleshoot and resolve issues in data pipelines and processing tasks. Ensure data security and compliance while working with large datasets. Monitor the performance of data pipelines and implement optimizations where necessary. Implement best practices for version control, testing, and CI/CD for data workflows. Required Skills & Qualifications: 5+ years of experience working with PySpark in a big data environment. Strong experience with Databricks for data engineering tasks. Proficiency in Python and its libraries for data processing and manipulation. Solid understanding of SQL for querying and working with data in relational databases. Experience working with cloud platforms like AWS , Azure , or GCP . Familiarity with Apache Spark for distributed data processing. Strong problem-solving skills and ability to troubleshoot complex data issues. Experience with version control systems like Git . Ability to work in a collaborative, fast-paced team environment.