Data Engineer

We are seeking an experienced Data Engineer to join our dynamic product development team. In this role, you will be responsible for designing, building, and optimizing data pipelines that ensure efficient data processing and insightful analytics. You will work collaboratively with cross-functional teams, including data scientists, software developers, and product managers, to transform raw data into actionable insights while adhering to best practices in data architecture, security, and scalability.

Role & responsibilities

* Design, build, and maintain scalable ETL processes to ingest, process, and store large datasets.

* Collaborate with cross-functional teams to integrate data from various sources, ensuring data consistency and quality.

* Leverage Microsoft Azure services for data storage, processing, and analytics, integrating with our CI/CD pipeline on Azure Repos.

* Continuously optimize data workflows for performance and scalability, identifying bottlenecks and implementing improvements.

* Deploy and monitor ML/GenAI models in production environments.

* Develop and enforce data quality standards, data validation checks, and ensure compliance with security and privacy policies.

* Work closely with backend developers (PHP/Node/Python) and DevOps teams to support seamless data operations and deployment.

* Stay current with industry trends and emerging technologies to continually enhance data strategies and methodologies.

Required Skills & Qualifications

* Minimum of 4+ years in data engineering or a related field.

* In depth understanding of streaming technologies like Kafka, Spark Streaming.

* Strong proficiency in SQL, Python, Spark SQL - data manipulation, data processing, and automation.

* Solid understanding of ETL/ELT frameworks, data pipeline design, data modelling, data warehousing and data governance principles.

* Must have in-depth knowledge of performance tuning/optimizing data processing jobs, debugging time consuming jobs.

* Proficient in Azure technologies like ADB, ADF, SQL (capability of writing complex SQL queries), PySpark, Python, Synapse, Fabric, Delta Tables, Unity CatLog.

* Deep understanding of cloud platforms (e.g., AWS, Azure, Google Cloud) and data warehousing solutions (e.g., Snowflake, Redshift, Big Query).

* Good knowledge of Agile, SDLC/CICD practices and tools with a good understanding of distributed systems.

* Proven ability to work effectively in agile/scrum teams, collaborating across disciplines.

* Excellent analytical, troubleshooting, problem-solving skills and attention to detail.

Preferred candidate profile

* Experience with NoSQL databases and big data processing frameworks e.g., Apache Spark.

* Knowledge of data visualization and reporting tools.

* Strong understanding of data security, governance, and compliance best practices.

* Effective communication skills with an ability to translate technical concepts to non-technical stakeholders.

* Knowledge of AI-OPS and LLM Data pipelines.

Why Join GenXAI?

* Innovative Environment: Work on transformative projects in a forward-thinking, collaborative setting.

* Career Growth: Opportunities for professional development and advancement within a rapidly growing company.

* Cutting-Edge Tools: Gain hands-on experience with industry-leading technologies and cloud platforms.

* Collaborative Culture: Join a diverse team where your expertise is valued, and your ideas make an impact.