Lead AI Data Engineer

Summary:

As a Data Engineer & AI Developer, you will play a key role in transforming complex data into clear, actionable insights. You will be responsible for building robust data pipelines, developing interactive dashboards, and integrating cutting-edge AI techniques to enhance our data processes and insights. This is a unique opportunity to blend expertise in data engineering and visualization with a forward-thinking approach to artificial intelligence.

Key Responsibilities:

• ETL/ELT Development: Design, build, and optimize scalable data pipelines using ETL/ELT processes to ingest, clean, and transform large datasets from various sources.
• AI-Powered Data Engineering: Implement and manage AI-driven ETL processes that use machine learning models for data quality checks, anomaly detection, and automated data transformation.
• Advanced Data Processing: Utilize Python and Apache Spark on the Databricks platform to perform large-scale data processing and feature engineering. Leverage tools like Delta Lake for efficient and reliable data management.
• Multi-Cloud Infrastructure: Architect and manage data solutions within the AWS and Azure ecosystem, leveraging services like S3 or ADLS for storage.
• AI/ML Integration: Work with data scientists to integrate machine learning models into production pipelines, ensuring data is prepared and delivered for model training and inference.
• Automated Workflows & Agentic CI/CD: Implement and maintain CI/CD pipelines using GitHub Actions, with a focus on creating agentic CI/CD workflows that automate tasks, trigger model retraining, or deploy data services based on dynamic triggers or data changes.
• Database Management: Write complex SQL queries and apply strong knowledge of database and data warehousing concepts to design and optimize data schemas.
• Software Engineering Principles: Adhere to best practices in software development, including writing clean, well-documented code, conducting code reviews, and designing for scalability and maintainability.

Required Skills:

• Proven experience in building and managing large-scale data pipelines.
• Strong proficiency in Python, including libraries like Pandas, NumPy, and PySpark.
• Hands-on expertise with Databricks and Apache Spark.
• Solid understanding of AWS services relevant to data engineering (S3, Glue, EMR).
• Solid understanding of Azure services relevant to data engineering (Data Lake, Azure HDInsight, Data Factory)
• Experience with GitHub and developing robust CI/CD pipelines.
• Proficiency in SQL and knowledge of relational and NoSQL databases.
• A strong foundation in software engineering principles.

AI Skills (Highly Desired):

• Experience with machine learning frameworks (e.g., scikit-learn, TensorFlow, PyTorch).
• Knowledge of AI/ML applications in data engineering, such as using models for data quality or feature selection.
• Familiarity with agentic systems and the ability to build agentic CI/CD pipelines that automate complex, multi-step processes.
• Experience with data governance and explainability in AI/ML pipelines.