Data Engineer

About Us:

Join our stealth-mode AI startup on a mission to revolutionize AI and data solutions. Headquartered in Hyderabad, we are a well-funded startup with a world-class team and a passion for innovation in AI, NLP, Computer Vision, and Speech Recognition.

Data Engineer

Role Overview:

Data Engineer

Key Responsibilities:

• Develop and maintain robust data pipelines using tools like

  PySpark

  ,

  Kafka

  , and

  Airflow

  .
• Design and optimize data workflows for high scalability and performance using

  Hadoop

  ,

  HDFS

  , and

  Hive

  .
• Integrate structured and unstructured data from diverse sources into a centralized platform.
• Leverage big data technologies for real-time processing and streaming using

  Spark Streaming

  and

  Nifi

  .
• Work on

  cloud-based platforms

  such as

  AWS

  ,

  Azure

  , and

  GCP

  to deploy and monitor scalable data solutions.
• Collaborate with AI/ML teams to deploy machine learning models using

  MLflow

  and integrate AI capabilities into data pipelines.
• Automate and monitor workflows to ensure seamless operations using

  CI/CD pipelines

  ,

  Kubernetes

  , and

  Docker

  .
• Implement data validation, performance testing, and troubleshooting of large-scale datasets.
• Prepare and share actionable insights through BI tools like

  Tableau

  and

  Grafana

  .

Required Skills and Qualifications:

• Education:

  Bachelor’s or Master’s degree in Computer Science, Data Engineering, or related fields.

• Experience:

  2 to 4 years in data engineering roles, working with big data ecosystems.

Technical Proficiency:

• Big Data Tools:

  Hadoop, HDFS, PySpark, Hive, Sqoop, Kafka, Spark Streaming, Airflow, Presto, Nifi.

• Cloud Platforms:

  AWS (Glue, S3, EMR), Azure (ADF, HDInsight), GCP (BigQuery, Pub/Sub).

• Programming Languages:

  Python, SQL, Scala.

• DevOps & Automation:

  Jenkins, Ansible, Kubernetes, Docker.

• Databases:

  MySQL, Oracle, HBase, Redis.

• Visualization Tools:

  Tableau, Grafana, Zeppelin.
• Knowledge of machine learning models, AI tools (e.g., TensorFlow, H2O), and feature engineering is a plus.
• Strong problem-solving skills with attention to detail and ability to manage multiple projects.
• Excellent communication and collaboration skills in a fast-paced environment.

What We Offer:

• Opportunity to work on innovative AI projects with a global impact.
• Collaborative work culture with access to cutting-edge technologies.