Platform Engineer

Job Responsibilities -

• Architect and implement a scalable, offline Data Lake for structured, semi-structured, and unstructured data in an on-premises, air-gapped environment.
• Collaborate with Data Engineers, Factory IT, and Edge Device teams to enable seamless data ingestion and retrieval across the platform.
• Integrate with upstream systems like MES, SCADA, and process tools to capture high-frequency manufacturing data efficiently.
• Monitor and maintain system health, including compute resources, storage arrays, disk I/O, memory usage, and network throughput.
• Optimize Data Lake performance via partitioning, deduplication, compression (Parquet/ORC), and implementing effective indexing strategies.
• Select, integrate, and maintain tools like Apache Hadoop, Spark, Hive, HBase, and custom ETL pipelines suitable for offline deployment.
• Build custom ETL workflows for bulk and incremental data ingestion using Python, Spark, and shell scripting.
• Implement data governance policies covering access control, retention periods, and archival procedures with security and compliance in mind.
• Establish and test backup, failover, and disaster recovery protocols specifically designed for offline environments.
• Document architecture designs, optimization routines, job schedules, and standard operating procedures (SOPs) for platform maintenance.
• Conduct root cause analysis for hardware failures, system outages, or data integrity issues.
• Drive system scalability planning for multi-fab or multi-site future expansions.

Essential Attributes (Tech-Stacks) -

• Hands-on experience designing and maintaining offline or air-gapped Data Lake environments.
• Deep understanding of Hadoop ecosystem tools: HDFS, Hive, Map-Reduce, HBase, YARN, zookeeper and Spark.
• Expertise in custom ETL design, large-scale batch and stream data ingestion.
• Strong scripting and automation capabilities using Bash and Python.
• Familiarity with data compression formats (ORC, Parquet) and ingestion frameworks (e.g., Flume).
• Working knowledge of message queues such as Kafka or RabbitMQ, with focus on integration logic.
• Proven experience in system performance tuning, storage efficiency, and resource optimization.

Qualifications

• BE/ ME in Computer science, Machine Learning, Electronics Engineering, Applied mathematics, Statistics.

Desired Experience Level -

• 4 Years relevant experience post Bachelors
• 2 Years relevant experience post Masters
• Experience with semiconductor industry is a plus