Senior Data Engineer

As a highly skilled and innovative Senior Data Engineer, your role will involve architecting and building scalable data pipelines, enabling intelligent data processing, and driving AI-powered analytics across enterprise datasets. You will primarily work with Azure Databricks, PySpark, and Snowflake, integrating AI tools such as ChatGPT, Copilot, or Databricks AI Functions. **Key Responsibilities:** - Design and develop scalable data pipelines using Azure Databricks, PySpark, and Delta Lake. - Implement and optimize data warehousing solutions using Snowflake. - Integrate AI capabilities into data workflows using tools like ChatGPT, Copilot, Databricks AI Functions, or Snowflake Copilot. - Collaborate with data scientists and analysts to enable AI-powered insights, including summarization, classification, and sentiment analysis. - Build and maintain ETL/ELT workflows, ensuring data quality, performance, and governance. - Work with stakeholders to understand business requirements and translate them into technical solutions. - Ensure security, compliance, and performance optimization across cloud data platforms. - Document architecture, workflows, and best practices for AI-enhanced data engineering. **Qualifications Required:** - 6 to 12 years of experience in data engineering, with expertise in Azure Databricks, PySpark, and Snowflake. - Deep understanding of Lakehouse architecture, Delta Lake, and data warehousing principles. - Proficiency in Python, SQL, and Spark for building scalable data pipelines and transformation workflows. - Experience integrating AI tools such as ChatGPT, Copilot, or Databricks AI Functions. - Familiarity with LangChain, pyspark-ai, or similar frameworks for embedding LLMs into data engineering tasks. - Knowledge of Snowflake Copilot, Snowpark, and AI-powered query generation. - Strong grasp of data governance, security, and metadata management. - Experience with CI/CD pipelines, DevOps practices, and infrastructure-as-code tools. - Ability to collaborate effectively with data scientists, analysts, and business stakeholders. - Excellent problem-solving, communication, and documentation skills. In addition to the technical aspects of the role, Persistent Ltd. offers a competitive salary and benefits package, a culture focused on talent development, opportunities to work with cutting-edge technologies, employee engagement initiatives, annual health check-ups, and insurance coverage for self, spouse, children, and parents. Persistent Ltd. is committed to fostering diversity and inclusion in the workplace, supporting hybrid work and flexible hours, and providing accessibility-friendly office setups for employees with disabilities. If you are a person with disabilities and have specific requirements, please inform during the application process or at any time during your employment. Join Persistent to unleash your full potential and be a part of an inclusive and values-driven work environment. As a highly skilled and innovative Senior Data Engineer, your role will involve architecting and building scalable data pipelines, enabling intelligent data processing, and driving AI-powered analytics across enterprise datasets. You will primarily work with Azure Databricks, PySpark, and Snowflake, integrating AI tools such as ChatGPT, Copilot, or Databricks AI Functions. **Key Responsibilities:** - Design and develop scalable data pipelines using Azure Databricks, PySpark, and Delta Lake. - Implement and optimize data warehousing solutions using Snowflake. - Integrate AI capabilities into data workflows using tools like ChatGPT, Copilot, Databricks AI Functions, or Snowflake Copilot. - Collaborate with data scientists and analysts to enable AI-powered insights, including summarization, classification, and sentiment analysis. - Build and maintain ETL/ELT workflows, ensuring data quality, performance, and governance. - Work with stakeholders to understand business requirements and translate them into technical solutions. - Ensure security, compliance, and performance optimization across cloud data platforms. - Document architecture, workflows, and best practices for AI-enhanced data engineering. **Qualifications Required:** - 6 to 12 years of experience in data engineering, with expertise in Azure Databricks, PySpark, and Snowflake. - Deep understanding of Lakehouse architecture, Delta Lake, and data warehousing principles. - Proficiency in Python, SQL, and Spark for building scalable data pipelines and transformation workflows. - Experience integrating AI tools such as ChatGPT, Copilot, or Databricks AI Functions. - Familiarity with LangChain, pyspark-ai, or similar frameworks for embedding LLMs into data engineering tasks. - Knowledge of Snowflake Copilot, Snowpark, and AI-powered query generation. - Strong grasp of data governance, security, and metadata management. - Experience with CI/CD pipelines, DevOps practices, and infrastructure-as-code tools. - Ability to collaborate effectively w